2025-10-31T10:10:24.829191Z :ReadMirrored INFO: Random seed for debugging is 1761905424829171
test_client.cpp: SetPath # /home/runner/.ya/build/build_root/vt0y/000f58/r3tmp/tmpHKt1P2/pdisk_1.dat
2025-10-31T10:10:25.991487Z node 3 :METADATA_PROVIDER WARN: log.cpp:841: fline=table_exists.cpp:54;actor=TTableExistsActor;event=undelivered;self_id=[3:7567326183293634353:2222];send_to=[0:7307199536658146131:7762515];
2025-10-31T10:10:26.079573Z node 4 :PQ_READ_PROXY DEBUG: caching_service.cpp:44: Direct read cache: : Created
2025-10-31T10:10:26.080982Z node 4 :KQP_PROXY WARN: kqp_finalize_script_service.cpp:123: [ScriptExecutions] [TKqpFinalizeScriptService] Failed to check script execution tables existence, got undelivered to scheme cache: ActorUnknown
2025-10-31T10:10:26.081044Z node 3 :PQ_READ_PROXY DEBUG: caching_service.cpp:44: Direct read cache: : Created
2025-10-31T10:10:26.084759Z node 3 :METADATA_PROVIDER ERROR: log.cpp:841: fline=accessor_snapshot_base.cpp:71;action=cannot detect path existence;path=//Root/.metadata/initialization/migrations;error=scheme_cache_undelivered_message;
2025-10-31T10:10:26.082299Z node 4 :METADATA_PROVIDER WARN: log.cpp:841: fline=table_exists.cpp:54;actor=TTableExistsActor;event=undelivered;self_id=[4:7567326184580670630:2273];send_to=[0:7307199536658146131:7762515];
2025-10-31T10:10:26.086781Z node 4 :METADATA_PROVIDER ERROR: log.cpp:841: fline=accessor_snapshot_base.cpp:71;action=cannot detect path existence;path=//Root/.metadata/initialization/migrations;error=scheme_cache_undelivered_message;
2025-10-31T10:10:26.116506Z node 3 :KQP_PROXY WARN: kqp_finalize_script_service.cpp:135: [ScriptExecutions] [TKqpFinalizeScriptService] Failed to check script execution tables existence, scheme status: LookupError, path: Root/.metadata/script_executions
2025-10-31T10:10:26.149582Z node 4 :KQP_PROXY WARN: kqp_finalize_script_service.cpp:135: [ScriptExecutions] [TKqpFinalizeScriptService] Failed to check script execution tables existence, scheme status: LookupError, path: Root/.metadata/script_executions
2025-10-31T10:10:26.349739Z node 4 :CONFIGS_DISPATCHER ERROR: configs_dispatcher.cpp:1229: Notification cookie mismatch for subscription [4:7567326180285703088:2073] 1761905425971914 != 1761905425971917
2025-10-31T10:10:26.351995Z node 3 :IMPORT WARN: schemeshard_import.cpp:307: Table profiles were not loaded
2025-10-31T10:10:26.393235Z node 3 :KQP_PROXY WARN: kqp_finalize_script_service.cpp:135: [ScriptExecutions] [TKqpFinalizeScriptService] Failed to check script execution tables existence, scheme status: LookupError, path: Root/.metadata/script_executions
2025-10-31T10:10:26.405535Z node 4 :KQP_PROXY WARN: kqp_finalize_script_service.cpp:135: [ScriptExecutions] [TKqpFinalizeScriptService] Failed to check script execution tables existence, scheme status: LookupError, path: Root/.metadata/script_executions
2025-10-31T10:10:26.407525Z node 3 :HIVE WARN: node_info.cpp:25: HIVE#72057594037968897 Node(3, (0,0,0,0)) VolatileState: Unknown -> Disconnected
2025-10-31T10:10:26.407605Z node 3 :HIVE WARN: node_info.cpp:25: HIVE#72057594037968897 Node(3, (0,0,0,0)) VolatileState: Disconnected -> Connecting
2025-10-31T10:10:26.409194Z node 3 :HIVE WARN: node_info.cpp:25: HIVE#72057594037968897 Node(4, (0,0,0,0)) VolatileState: Unknown -> Disconnected
2025-10-31T10:10:26.409252Z node 3 :HIVE WARN: node_info.cpp:25: HIVE#72057594037968897 Node(4, (0,0,0,0)) VolatileState: Disconnected -> Connecting
2025-10-31T10:10:26.424382Z node 3 :HIVE WARN: node_info.cpp:25: HIVE#72057594037968897 Node(3, (0,0,0,0)) VolatileState: Connecting -> Connected
2025-10-31T10:10:26.434984Z node 3 :HIVE WARN: hive_impl.cpp:811: HIVE#72057594037968897 Handle TEvInterconnect::TEvNodeConnected, NodeId 4 Cookie 4
2025-10-31T10:10:26.436660Z node 3 :HIVE WARN: node_info.cpp:25: HIVE#72057594037968897 Node(4, (0,0,0,0)) VolatileState: Connecting -> Connected
TServer::EnableGrpc on GrpcPort 3655, node 3
2025-10-31T10:10:26.657796Z node 3 :NET_CLASSIFIER WARN: net_classifier.cpp:190: distributable config is empty, broken or outdated, will use file: /home/runner/.ya/build/build_root/vt0y/000f58/r3tmp/yandexNJzwSX.tmp
2025-10-31T10:10:26.657823Z node 3 :NET_CLASSIFIER WARN: net_classifier.cpp:196: will try to initialize from file: /home/runner/.ya/build/build_root/vt0y/000f58/r3tmp/yandexNJzwSX.tmp
2025-10-31T10:10:26.657953Z node 3 :NET_CLASSIFIER WARN: net_classifier.cpp:202: successfully initialized from file: /home/runner/.ya/build/build_root/vt0y/000f58/r3tmp/yandexNJzwSX.tmp
2025-10-31T10:10:26.658107Z node 3 :NET_CLASSIFIER ERROR: net_classifier.cpp:228: got bad distributable configuration
2025-10-31T10:10:26.717767Z INFO: TTestServer started on Port 16049 GrpcPort 3655
2025-10-31T10:10:26.810274Z node 4 :KQP_PROXY WARN: kqp_finalize_script_service.cpp:135: [ScriptExecutions] [TKqpFinalizeScriptService] Failed to check script execution tables existence, scheme status: PathErrorUnknown, path: Root/.metadata/script_executions
2025-10-31T10:10:26.946711Z node 3 :KQP_PROXY WARN: kqp_finalize_script_service.cpp:135: [ScriptExecutions] [TKqpFinalizeScriptService] Failed to check script execution tables existence, scheme status: PathErrorUnknown, path: Root/.metadata/script_executions
TClient is connected to server localhost:16049
PQClient connected to localhost:3655
WaitRootIsUp 'Root'...
TClient::Ls request: Root
2025-10-31T10:10:26.997219Z node 3 :TX_CONVEYOR ERROR: log.cpp:841: fline=service.h:53;problem=unexpected event for task executor;ev_type=NActors::TEvents::TEvWakeup;
TClient::Ls response: Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "Root"
    PathId: 1
    SchemeshardId: 72057594046644480
    PathType: EPathTypeDir
    CreateFinished: true
    CreateTxId: 1
    CreateStep: 0
    ParentPathId: 1
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 1
      SubDomainVersion: 0
      SecurityStateVersion: 0
    }
    ChildrenExist: false
  }
  Children {
    Name: ".sys"
    PathId: 18446744073709551615
    SchemeshardId: 72057594046644480
    PathType: EPathTypeDir
    CreateFinished: true
    CreateTxId: 0
    CreateStep: 0
    ParentPathId: 18446744073709551615
  }
  DomainDescription {
    SchemeShardId_Depricated: 72057594046644480
    PathId_Depricated: 1
    ProcessingParams {
      Version: 0
      Pl...
(TRUNCATED)
WaitRootIsUp 'Root' success.
2025-10-31T10:10:27.011580Z node 3 :FLAT_TX_SCHEMESHARD WARN: schemeshard__operation.cpp:183: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpAlterSubDomain, opId: 281474976710657:0, at schemeshard:  72057594046644480, first GetDB called at:  (GetDB first called at /home/runner/actions_runner/_work/ydb/ydb/ydb/core/tx/schemeshard/schemeshard__operation_alter_subdomain.cpp:311)
waiting...
2025-10-31T10:10:27.020166Z node 4 :TX_CONVEYOR ERROR: log.cpp:841: fline=service.h:53;problem=unexpected event for task executor;ev_type=NActors::TEvents::TEvWakeup;
waiting...
waiting...
waiting...
2025-10-31T10:10:30.926003Z node 3 :METADATA_PROVIDER ERROR: log.cpp:841: fline=table_exists.cpp:59;actor=TTableExistsActor;event=timeout;self_id=[3:7567326183293634353:2222];send_to=[0:7307199536658146131:7762515];
2025-10-31T10:10:30.926097Z node 3 :METADATA_PROVIDER ERROR: log.cpp:841: fline=accessor_snapshot_base.cpp:71;action=cannot detect path existence;path=//Root/.metadata/initialization/migrations;error=timeout;
2025-10-31T10:10:31.083600Z node 4 :METADATA_PROVIDER ERROR: log.cpp:841: fline=table_exists.cpp:59;actor=TTableExistsActor;event=timeout;self_id=[4:7567326184580670630:2273];send_to=[0:7307199536658146131:7762515];
2025-10-31T10:10:31.083682Z node 4 :METADATA_PROVIDER ERROR: log.cpp:841: fline=accessor_snapshot_base.cpp:71;action=cannot detect path existence;path=//Root/.metadata/initialization/migrations;error=timeout;
2025-10-31T10:10:31.497476Z node 4 :KQP_WORKLOAD_SERVICE WARN: scheme_actors.cpp:225: [WorkloadService] [TPoolFetcherActor] ActorId: [4:7567326206055507256:2304], DatabaseId: /Root, PoolId: default, Failed to fetch pool info, NOT_FOUND, issues: { <main>: Error: Resource pool default not found or you don't have access permissions }
2025-10-31T10:10:31.497549Z node 4 :KQP_WORKLOAD_SERVICE WARN: scheme_actors.cpp:225: [WorkloadService] [TPoolFetcherActor] ActorId: [4:7567326206055507224:2301], DatabaseId: /Root, PoolId: default, Failed to fetch pool info, NOT_FOUND, issues: { <main>: Error: Resource pool default not found or you don't have access permissions }
2025-10-31T10:10:31.497622Z node 4 :KQP_WORKLOAD_SERVICE WARN: kqp_workload_service.cpp:263: [WorkloadService] [Service] Failed to fetch pool default, DatabaseId: /Root, status: NOT_FOUND, issues: { <main>: Error: Resource pool default not found or you don't have access permissions }
2025-10-31T10:10:31.505294Z node 4 :KQP_WORKLOAD_SERVICE WARN: scheme_actors.cpp:225: [WorkloadService] [TPoolFetcherActor] ActorId: [4:7567326206055507262:2306], DatabaseId: /Root, PoolId: default, Failed to fetch pool info, NOT_FOUND, issues: { <main>: Error: Resource pool default not found or you don't have access permissions }
2025-10-31T10:10:31.505430Z node 4 :KQP_WORKLOAD_SERVICE WARN: kqp_workload_service.cpp:263: [WorkloadService] [Service] Failed to fetch pool default, DatabaseId: /Root, status: NOT_FOUND, issues: { <main>: Error: Resource pool default not found or you don't have access permissions }
2025-10-31T10:10:31.515612Z node 3 :FLAT_TX_SCHEMESHARD WARN: schemeshard__operation.cpp:183: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpCreateResourcePool, opId: 281474976720657:3, at schemeshard:  72057594046644480, first GetDB called at:  (GetDB first called at /home/runner/actions_runner/_work/ydb/ydb/ydb/core/tx/schemeshard/schemeshard__operation_create_resource_pool.cpp:179)
2025-10-31T10:10:31.550695Z node 4 :KQP_WORKLOAD_SERVICE WARN: helpers.h:73: [WorkloadService] [TPoolCreatorActor] ActorId: [4:7567326206055507261:2305], DatabaseId: /Root, PoolId: default, Scheduled retry for error: { <main>: Error: Transaction 281474976720657 completed, doublechecking }
2025-10-31T10:10:31.618908Z node 4 :TX_PROXY ERROR: schemereq.cpp:593: Actor# [4:7567326206055507291:2144] txid# 281474976720658, issues: { message: "Check failed: path: \'/Root/.metadata/workload_manager/pools/default\', error: path exist, request accepts it (id: [OwnerId: 72057594046644480, LocalPathId: 8], type: EPathTypeResourcePool, state: EPathStateNoChanges)" severity: 1 }
2025-10-31T10:10:31.666170Z node 4 :KQP_COMPILE_ACTOR ERROR: kqp_compile_actor.cpp:570: Compilation failed, self: [4:7567326206055507298:2310], status: SCHEME_ERROR, issues: <main>: Error: Type annotation, code: 1030
    <main>:3:16: Error: At function: KiReadTable!
        <main>:3:16: Error: Cannot find table 'db.[/Root/PQ/Config/V2/Cluster]' because it does not exist or you do not have access permissions. Please check correctness of table path and user permissions., code: 2003
2025-10-31T10:10:31.666407Z node 4 :KQP_SESSION WARN: kqp_session_actor.cpp:2500: SessionId: ydb://session/3?node_id=4&id=NDRhZjEzZDItMjNmMmVhMDYtNGEwYjQ3MzItYmU4M2UxYzk=, ActorId: [4:7567326206055507220:2298], ActorState: ExecuteState, TraceId: 01k8wvymy0exdpc28yywka82n1, ReplyQueryCompileError, status SCHEME_ERROR remove tx with tx_id: 
2025-10-31T10:10:31.666840Z node 4 :PERSQUEUE_CLUSTER_TRACKER ERROR: cluster_tracker.cpp:167: failed to list clusters: { Response { QueryIssues { message: "Type annotation" issue_code: 1030 severity: 1 issues { position { row: 3 column: 16 } message: "At function: KiReadTable!" end_position { row: 3 column: 16 } severity: 1 issues { position { row: 3 column: 16 } message: "Cannot find table \'db.[/Root/PQ/Config/V2/Cluster]\' because it does not exist or you do not have access permissions. Please check correctness of table path and user permissions." end_position { row: 3 column: 16 } issue_code: 2003 severity: 1 } } } TxMeta { } } YdbStatus: SCHEME_ERROR ConsumedRu: 1 }
2025-10-31T10:10:31.689213Z node 3 :KQP_COMPILE_ACTOR ERROR: kqp_compile_actor.cpp:570: Compilation failed, self: [3:7567326209063439041:2335], status: SCHEME_ERROR, issues: <main>: Error: Type annotation, code: 1030
    <main>:3:16: Error: At function: KiReadTable!
        <main>:3:16: Error: Cannot find table 'db.[/Root/PQ/Config/V2/Cluster]' because it does not exist or you do not have access permissions. Please check correctness of table path and user permissions., code: 2003
2025-10-31T10:10:31.691967Z node 3 :KQP_SESSION WARN: kqp_session_actor.cpp:2500: SessionId: ydb://session/3?node_id=3&id=OTExNDk2M2MtMzE4N2MxN2ItNDY0NGNmY2EtMmNhNzIwYjg=, ActorId: [3:7567326209063438991:2327], ActorState: ExecuteState, TraceId: 01k8wvyn3874xz7kvag1079fn7, ReplyQueryCompileError, status SCHEME_ERROR remove tx with tx_id: 
2025-10-31T10:10:31.692416Z node 3 :PERSQUEUE_CLUSTER_TRACKER ERROR: cluster_tracker.cpp:167: failed to list clusters: { Response { QueryIssues { message: "Type annotation" issue_code: 1030 severity: 1 issues { position { row: 3 column: 16 } message: "At function: KiReadTable!" end_position { row: 3 column: 16 } severity: 1 issues { position { row: 3 column: 16 } message: "Cannot find table \'db.[/Root/PQ/Config/V2/Cluster]\' because it does not exist or you do not have access permissions. Please check correctness of table path and user permissions." end_position { row: 3 column: 16 } issue_code: 2003 severity: 1 } } } TxMeta { } } YdbStatus: SCHEME_ERROR ConsumedRu: 1 }
2025-10-31T10:10:31.722062Z node 3 :FLAT_TX_SCHEMESHARD WARN: schemeshard__operation.cpp:183: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpCreateTable, opId: 281474976710661:0, at schemeshard:  72057594046644480, first GetDB called at:  (GetDB first called at /home/runner/actions_runner/_work/ydb/ydb/ydb/core/tx/schemeshard/schemeshard__operation_create_table.cpp:690)
2025-10-31T10:10:31.879433Z node 3 :FLAT_TX_SCHEMESHARD WARN: schemeshard__operation.cpp:183: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpCreateTable, opId: 281474976710662:0, at schemeshard:  72057594046644480, first GetDB called at:  (GetDB first called at /home/runner/actions_runner/_work/ydb/ydb/ydb/core/tx/schemeshard/schemeshard__operation_create_table.cpp:690)
2025-10-31T10:10:32.095321Z node 3 :FLAT_TX_SCHEMESHARD WARN: schemeshard__operation.cpp:183: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpCreateTable, opId: 281474976710663:0, at schemeshard:  72057594046644480, first GetDB called at:  (GetDB first called at /home/runner/actions_runner/_work/ydb/ydb/ydb/core/tx/schemeshard/schemeshard__operation_create_table.cpp:690)
=== Init DC: UPSERT INTO `/Root/PQ/Config/V2/Cluster` (name, balancer, local, enabled, weight) VALUES ("dc1", "localhost:3655", true, true, 1000), ("dc2", "dc2.logbroker.yandex.net", false, false, 1000);
2025-10-31T10:10:32.550659Z node 3 :KQP_EXECUTER ERROR: kqp_planner.cpp:123: TxId: 281474976710664. Ctx: { TraceId: 01k8wvynpt7ek444akqz0w4mvb, Database: , SessionId: ydb://session/3?node_id=3&id=YzFiOTU5ZWEtN2MxMjVmZWMtZmJkMTc2Yi01YTBkOWE4, PoolId: default, DatabaseId: /Root}. Database not set, use /Root
=== CheckClustersList. Subcribe to ClusterTracker from [3:7567326213358406734:2977] 
=== CheckClustersList. Ok
2025-10-31T10:10:37.697221Z node 3 :FLAT_TX_SCHEMESHARD WARN: schemeshard__operation.cpp:183: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpCreateTable, opId: 281474976710669:0, at schemeshard:  72057594046644480, first GetDB called at:  (GetDB first called at /home/runner/actions_runner/_work/ydb/ydb/ydb/core/tx/schemeshard/schemeshard__operation_create_table.cpp:690)
waiting...
PQ Client: create topic: rt3.dc1--test-topic with 1 partitions
CallPersQueueGRPC request to localhost:3655
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic"
  }
}
2025-10-31T10:10:37.803580Z node 3 :PERSQUEUE INFO: msgbus_server_persqueue.cpp:1524: proxy answer <none>
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
CallPersQueueGRPC request to localhost:3655
MetaRequest {
  CmdCreateTopic {
    Topic: "rt3.dc1--test-topic"
    NumPartitions: 1
    Config {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      Consumers {
        Name: "user"
        ReadFromTimestampsMs: 0
        FormatVersion: 0
        Version: 0
      }
    }
  }
}
CallPersQueueGRPC response:
Status: 129
ProxyErrorCode: 53
SchemeStatus: 1
FlatTxId {
  TxId: 281474976710670
  SchemeShardTabletId: 72057594046644480
  PathId: 13
}
ErrorCode: OK
AddTopic: rt3.dc1--test-topic
===Run query:``DECLARE $version as Int64; DECLARE $path AS Utf8; DECLARE $cluster as Utf8; UPSERT INTO `/Root/PQ/Config/V2/Topics` (path, dc) VALUES ($path, $cluster); UPSERT INTO `/Root/PQ/Config/V2/Versions` (name, version) VALUES ("Topics", $version);`` with topic = test-topic, dc = dc1
2025-10-31T10:10:37.852844Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037893][] pipe [3:7567326234833243618:3240] connected; active server actors: 1
2025-10-31T10:10:37.853130Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1519: [72075186224037893][rt3.dc1--test-topic] updating configuration. Deleted partitions []. Added partitions [0]
2025-10-31T10:10:37.855419Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer.cpp:971: [72075186224037893][rt3.dc1--test-topic] Discovered subdomain [OwnerId: 72057594046644480, LocalPathId: 1] state, outOfSpace = 0 at RB 72075186224037893
2025-10-31T10:10:37.856822Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer.cpp:140: [72075186224037893][rt3.dc1--test-topic] BALANCER INIT DONE for rt3.dc1--test-topic: (0, 72075186224037892) 
2025-10-31T10:10:37.866340Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2960: [PQ: 72075186224037892] Handle TEvInterconnect::TEvNodeInfo
2025-10-31T10:10:37.874315Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2992: [PQ: 72075186224037892] Transactions request. From tx_00000000000000000000, To tx_18446744073709551615
2025-10-31T10:10:37.874530Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:787: [PQ: 72075186224037892] no config, start with empty partitions and default config
2025-10-31T10:10:37.874551Z node 4 :PERSQUEUE NOTICE: pq_impl.cpp:913: [PQ: 72075186224037892] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-10-31T10:10:37.874586Z node 4 :PERSQUEUE INFO: pq_impl.cpp:612: [PQ: 72075186224037892] doesn't have tx writes info
2025-10-31T10:10:37.874876Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037892] server connected, pipe [3:7567326234833243635:2431], now have 1 active actors on pipe
2025-10-31T10:10:37.874901Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037892] server connected, pipe [3:7567326234833243617:3239], now have 1 active actors on pipe
2025-10-31T10:10:37.875245Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer.cpp:404: [72075186224037893][rt3.dc1--test-topic] TEvClientConnected TabletId 72075186224037892, NodeId 4, Generation 1
2025-10-31T10:10:37.876124Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:37.883464Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:37.895639Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037892] server connected, pipe [3:7567326234833243675:3284], now have 1 active actors on pipe
2025-10-31T10:10:37.896207Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:37.899757Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:37.900655Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:81: [rt3.dc1--test-topic:0:Initializer] Start initializing step TInitConfigStep
2025-10-31T10:10:37.900706Z node 4 :PERSQUEUE DEBUG: partition.cpp:1295: [72075186224037892][Partition][0][StateInit] HandleOnInit TEvPQ::TEvProposePartitionConfig
2025-10-31T10:10:37.901020Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:81: [rt3.dc1--test-topic:0:Initializer] Start initializing step TInitInternalFieldsStep
2025-10-31T10:10:37.901247Z node 4 :PERSQUEUE INFO: partition_init.cpp:1101: [72075186224037892][Partition][0][StateInit] bootstrapping 0 [4:7567326231825311516:2385]
2025-10-31T10:10:37.904850Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:81: [rt3.dc1--test-topic:0:Initializer] Start initializing step TInitFieldsStep
2025-10-31T10:10:37.904886Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:61: [rt3.dc1--test-topic:0:Initializer] Initializing completed.
2025-10-31T10:10:37.904907Z node 4 :PERSQUEUE INFO: partition.cpp:707: [72075186224037892][Partition][0][StateInit] init complete for topic 'rt3.dc1--test-topic' partition 0 generation 1 [4:7567326231825311516:2385]
2025-10-31T10:10:37.904933Z node 4 :PERSQUEUE DEBUG: partition.cpp:721: [72075186224037892][Partition][0][StateInit] SYNC INIT topic rt3.dc1--test-topic partitition 0 so 0 endOffset 0 Head Offset 0 PartNo 0 PackedSize 0 count 0 nextOffset 0 batches 0
2025-10-31T10:10:37.904966Z node 4 :PERSQUEUE DEBUG: partition_mlp.cpp:83: [72075186224037892][Partition][0][StateIdle] Initializing MLP Consumers: 0
2025-10-31T10:10:37.906182Z node 4 :PERSQUEUE DEBUG: partition.cpp:4256: [72075186224037892][Partition][0][StateIdle] Process pending events. Count 1
2025-10-31T10:10:37.906222Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:37.906319Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:37.906337Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037892][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:37.906355Z node 4 :PERSQUEUE DEBUG: partition.cpp:2301: [72075186224037892][Partition][0][StateIdle] Still pending for come commits
2025-10-31T10:10:37.906368Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:37.906378Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:37.906387Z node 4 :PERSQUEUE DEBUG: partition.cpp:2301: [72075186224037892][Partition][0][StateIdle] Still pending for come commits
2025-10-31T10:10:37.906414Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:926: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-10-31T10:10:37.906443Z node 4 :PERSQUEUE DEBUG: partition_mlp.cpp:60: [72075186224037892][Partition][0][StateIdle] Process MLP pending events. Count 0
2025-10-31T10:10:37.906692Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:37.906834Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:183: [72075186224037892][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:37.914332Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:37.914464Z node 4 :PERSQUEUE DEBUG: partition.cpp:1399: [72075186224037892][Partition][0][StateIdle] Handle TEvPQ::TEvTxCommit Step 1761905437936, TxId 281474976710670
2025-10-31T10:10:37.914484Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:37.914497Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:37.914606Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:37.914789Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037892][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:37.918636Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037892][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:37.918796Z node 4 :PERSQUEUE DEBUG: partition_mlp.cpp:83: [72075186224037892][Partition][0][StateIdle] Initializing MLP Consumers: 0
2025-10-31T10:10:37.918889Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037892][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:37.918911Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:37.918924Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:37.918935Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:37.918962Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:183: [72075186224037892][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:37.919290Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:398: [PQ: 72075186224037892] Apply new config PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Version: 0 Generation: 0 } MonitoringProjectId: ""
2025-10-31T10:10:37.919364Z node 4 :PERSQUEUE NOTICE: pq_impl.cpp:913: [PQ: 72075186224037892] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-10-31T10:10:37.919605Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:37.937491Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:37.937590Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:37.942718Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:37.970060Z node 3 :KQP_EXECUTER ERROR: kqp_planner.cpp:123: TxId: 281474976710671. Ctx: { TraceId: 01k8wvyv4z86ka445fj3g6xb5w, Database: , SessionId: ydb://session/3?node_id=3&id=YTM3YTRjMTctMTM3NTcwYzctNTJkYjE3YmEtZTdmNjQ4ZDg=, PoolId: default, DatabaseId: /Root}. Database not set, use /Root
===Query complete
TClient::Ls request: /Root/PQ/rt3.dc1--test-topic
2025-10-31T10:10:38.005429Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.005460Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.005475Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
TClient::Ls response: Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic"
    PathId: 13
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976710670
    CreateStep: 1761905437936
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic"
    PathId: 13
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts...
(TRUNCATED)
GetTopicVersionFromPath:  record Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic"
    PathId: 13
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976710670
    CreateStep: 1761905437936
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic"
    PathId: 13
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      YdbDatabasePath: "/Root"
      Consumers {
        Name: "user"
        ReadFromTimestampsMs: 0
        FormatVersion: 0
        Version: 0
      }
    }
    Partitions {
      PartitionId: 0
      TabletId: 72075186224037892
      Status: Active
    }
    AlterVersion: 1
    BalancerTabletID: 72075186224037893
    NextPartitionId: 1
  }
  DomainDescription {
    SchemeShardId_Depricated: 72057594046644480
    PathId_Depricated: 1
    ProcessingParams {
      Version: 1
      PlanResolution: 7
      Coordinators: 72057594046316545
      TimeCastBucketsPerMediator: 2
      Mediators: 72057594046382081
    }
    DomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    PathsInside: 12
    PathsLimit: 10000
    ShardsInside: 6
    ShardsLimit: 200000
    ResourcesDomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    DiskSpaceUsage {
      Tables {
        TotalSize: 0
        DataSize: 0
        IndexSize: 0
      }
      Topics {
        ReserveSize: 0
        AccountSize: 0
        DataSize: 0
        UsedReserveSize: 0
      }
    }
    PQPartitionsInside: 1
    PQPartitionsLimit: 1000000
    SchemeLimits {
      MaxDepth: 32
      MaxPaths: 10000
      MaxChildrenInDir: 100000
      MaxAclBytesSize: 10240
      MaxTableColumns: 200
      MaxTableColumnNameLength: 255
      MaxTableKeyColumns: 30
      MaxTableIndices: 20
      MaxShards: 200000
      MaxShardsInPath: 35000
      MaxConsistentCopyTargets: 10000
      MaxPathElementLength: 255
      ExtraPathSymbolsAllowed: "!\"#$%&\'()*+,-.:;<=>?@[\\]^_`{|}~"
      MaxPQPartitions: 1000000
      MaxTableCdcStreams: 5
      MaxExports: 10
      MaxImports: 10
      MaxColumnTableColumns: 10000
    }
  }
}
Path: "/Root/PQ/rt3.dc1--test-topic"
 name rt3.dc1--test-topic version1
CallPersQueueGRPC request to localhost:3655
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic"
  }
}
2025-10-31T10:10:38.054318Z node 3 :PERSQUEUE INFO: msgbus_server_persqueue.cpp:1524: proxy answer <none>
CallPersQueueGRPC response:
Status: 1
ErrorCode: OK
MetaResponse {
  CmdGetTopicMetadataResult {
    TopicInfo {
      Topic: "rt3.dc1--test-topic"
      NumPartitions: 1
      Config {
        PartitionConfig {
          LifetimeSeconds: 86400
          LowWatermark: 8388608
          SourceIdLifetimeSeconds: 86400
          WriteSpeedInBytesPerSecond: 20000000
          BurstSize: 20000000
          SourceIdMaxCounts: 6000000
        }
        Version: 1
        LocalDC: true
        Codecs {
          Ids: 0
          Ids: 1
          Ids: 2
          Codecs: "raw"
          Codecs: "gzip"
          Codecs: "lzop"
        }
        TopicPath: "/Root/PQ/rt3.dc1--test-topic"
        YdbDatabasePath: "/Root"
        Consumers {
          Name: "user"
          ReadFromTimestampsMs: 0
          FormatVersion: 0
          Version: 0
        }
      }
      ErrorCode: OK
    }
  }
}
=== Topic created, have version: 1
2025-10-31T10:10:38.065658Z :DEBUG: [] MessageGroupId [src] SessionId [] Write session: try to update token
2025-10-31T10:10:38.066228Z :INFO: [] MessageGroupId [src] SessionId [] Write session: Do CDS request
2025-10-31T10:10:38.066274Z :INFO: [] MessageGroupId [src] SessionId [] Start write session. Will connect to endpoint: localhost:3655
2025-10-31T10:10:38.073436Z :DEBUG: [] MessageGroupId [src] SessionId [] Write session: send init request: init_request { topic: "test-topic" message_group_id: "src" }
2025-10-31T10:10:38.074182Z node 3 :PQ_WRITE_PROXY DEBUG: grpc_pq_write.h:107: new grpc connection
2025-10-31T10:10:38.074213Z node 3 :PQ_WRITE_PROXY DEBUG: grpc_pq_write.h:141: new session created cookie 1
2025-10-31T10:10:38.075177Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 1 sessionId:  grpc read done: success: 1 data: init_request { topic: "test-topic" message_group_id: "src" }
2025-10-31T10:10:38.075302Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:447: session request cookie: 1 topic: "test-topic" message_group_id: "src" from ipv6:[::1]:33442
2025-10-31T10:10:38.075323Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:1538: write session:  cookie=1 sessionId= userAgent="pqv1 server" ip=ipv6:[::1]:33442 proto=v1  topic=test-topic durationSec=0
2025-10-31T10:10:38.075335Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:571: init check schema
2025-10-31T10:10:38.076981Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:632: session v1 cookie: 1 sessionId:  describe result for acl check
2025-10-31T10:10:38.077079Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:62: TTableHelper SelectQuery: --!syntax_v1
DECLARE $Hash AS Uint32; DECLARE $Topic AS Utf8; DECLARE $SourceId AS Utf8;
SELECT Partition, CreateTime, AccessTime, SeqNo FROM `/Root/PQ/SourceIdMeta2` WHERE Hash == $Hash AND Topic == $Topic AND SourceId == $SourceId;
2025-10-31T10:10:38.077085Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:63: TTableHelper UpdateQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;DECLARE $SeqNo AS Uint64;
UPSERT INTO `/Root/PQ/SourceIdMeta2` (Hash, Topic, SourceId, CreateTime, AccessTime, Partition, SeqNo) VALUES ($Hash, $Topic, $SourceId, $CreateTime, $AccessTime, $Partition, $SeqNo);
2025-10-31T10:10:38.077113Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:64: TTableHelper UpdateAccessTimeQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;
UPDATE `/Root/PQ/SourceIdMeta2` SET AccessTime = $AccessTime WHERE Hash = $Hash AND Topic = $Topic AND SourceId = $SourceId AND Partition = $Partition;
2025-10-31T10:10:38.077129Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:111: TPartitionChooser [3:7567326239128211046:2449] (SourceId=src, PreferedPartition=(NULL)) StartKqpSession
2025-10-31T10:10:38.079318Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:142: TPartitionChooser [3:7567326239128211046:2449] (SourceId=src, PreferedPartition=(NULL)) Select from the table
2025-10-31T10:10:38.105875Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.105897Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.105907Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.206157Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.206188Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.206202Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.258026Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__old_chooser_actor.h:67: TPartitionChooser [3:7567326239128211046:2449] (SourceId=src, PreferedPartition=(NULL)) RequestPQRB
2025-10-31T10:10:38.258808Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326239128211085:2449] connected; active server actors: 1
2025-10-31T10:10:38.258975Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__old_chooser_actor.h:80: TPartitionChooser [3:7567326239128211046:2449] (SourceId=src, PreferedPartition=(NULL)) Received partition 0 from PQRB for SourceId=src
2025-10-31T10:10:38.258996Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:174: TPartitionChooser [3:7567326239128211046:2449] (SourceId=src, PreferedPartition=(NULL)) Update the table
2025-10-31T10:10:38.264250Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1659: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326239128211085:2449] disconnected.
2025-10-31T10:10:38.264281Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1670: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326239128211085:2449] disconnected; active server actors: 1
2025-10-31T10:10:38.264302Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1695: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326239128211085:2449] disconnected no session
2025-10-31T10:10:38.307571Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.307605Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.307620Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.371914Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:183: TPartitionChooser [3:7567326239128211046:2449] (SourceId=src, PreferedPartition=(NULL)) HandleUpdate PartitionPersisted=0 Status=SUCCESS
2025-10-31T10:10:38.371948Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:305: TPartitionChooser [3:7567326239128211046:2449] (SourceId=src, PreferedPartition=(NULL)) ReplyResult: Partition=0, SeqNo=(NULL)
2025-10-31T10:10:38.371963Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:268: TPartitionChooser [3:7567326239128211046:2449] (SourceId=src, PreferedPartition=(NULL)) Start idle
2025-10-31T10:10:38.371991Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:694: ProceedPartition. session cookie: 1 sessionId:  partition: 0 expectedGeneration: (NULL)
2025-10-31T10:10:38.373738Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037892] server connected, pipe [3:7567326239128211109:2449], now have 1 active actors on pipe
2025-10-31T10:10:38.374035Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:822: TPartitionWriter 72075186224037892 (partition=0) TEvClientConnected Status OK, TabletId: 72075186224037892, NodeId 4, Generation: 1
2025-10-31T10:10:38.374369Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:38.374412Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:38.374519Z node 4 :PERSQUEUE INFO: ownerinfo.cpp:30: new Cookie src|5c23461d-16650d26-6e75d6e0-6c0635f_0 generated for partition 0 topic 'rt3.dc1--test-topic' owner src
2025-10-31T10:10:38.374580Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.374601Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.374621Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037892][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:38.374659Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.374698Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:35: [72075186224037892][Partition][0][StateIdle] TPartition::ReplyOwnerOk. Partition: 0
2025-10-31T10:10:38.374768Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:38.375313Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:38.375341Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:38.375406Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:38.375686Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:871: session inited cookie: 1 partition: 0 MaxSeqNo: 0 sessionId: src|5c23461d-16650d26-6e75d6e0-6c0635f_0
2025-10-31T10:10:38.376767Z :INFO: [] MessageGroupId [src] SessionId [] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1761905438376 BytesWritten: 0 MessagesWritten: 0 BytesWrittenCompressed: 0 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-10-31T10:10:38.376891Z :INFO: [] MessageGroupId [src] SessionId [] Write session established. Init response: session_id: "src|5c23461d-16650d26-6e75d6e0-6c0635f_0" topic: "test-topic" cluster: "dc1" supported_codecs: CODEC_RAW supported_codecs: CODEC_GZIP supported_codecs: CODEC_LZOP
2025-10-31T10:10:38.377081Z :INFO: [] MessageGroupId [src] SessionId [src|5c23461d-16650d26-6e75d6e0-6c0635f_0] Write session: close. Timeout = 0 ms
2025-10-31T10:10:38.377144Z :INFO: [] MessageGroupId [src] SessionId [src|5c23461d-16650d26-6e75d6e0-6c0635f_0] Write session will now close
2025-10-31T10:10:38.377205Z :DEBUG: [] MessageGroupId [src] SessionId [src|5c23461d-16650d26-6e75d6e0-6c0635f_0] Write session: aborting
2025-10-31T10:10:38.377633Z :INFO: [] MessageGroupId [src] SessionId [src|5c23461d-16650d26-6e75d6e0-6c0635f_0] Write session: gracefully shut down, all writes complete
2025-10-31T10:10:38.377680Z :DEBUG: [] MessageGroupId [src] SessionId [src|5c23461d-16650d26-6e75d6e0-6c0635f_0] Write session: destroy
2025-10-31T10:10:38.379096Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2746: [PQ: 72075186224037892] server disconnected, pipe [3:7567326239128211109:2449] destroyed
2025-10-31T10:10:38.379142Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:138: [72075186224037892][Partition][0][StateIdle] TPartition::DropOwner.
2025-10-31T10:10:38.379170Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.379185Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.379198Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
PQ Client: create topic: rt3.dc1--test-topic-mirrored-from-dc2 with 1 partitions
CallPersQueueGRPC request to localhost:3655
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc2"
  }
}
2025-10-31T10:10:38.378310Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 1 sessionId: src|5c23461d-16650d26-6e75d6e0-6c0635f_0 grpc read done: success: 0 data: 
2025-10-31T10:10:38.378331Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:256: session v1 cookie: 1 sessionId: src|5c23461d-16650d26-6e75d6e0-6c0635f_0 grpc read failed
2025-10-31T10:10:38.378396Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:824: session v1 closed cookie: 1 sessionId: src|5c23461d-16650d26-6e75d6e0-6c0635f_0
2025-10-31T10:10:38.378422Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:304: session v1 cookie: 1 sessionId: src|5c23461d-16650d26-6e75d6e0-6c0635f_0 is DEAD
2025-10-31T10:10:38.378692Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::TEvents::TEvPoison
2025-10-31T10:10:38.405699Z node 3 :PERSQUEUE INFO: msgbus_server_persqueue.cpp:1524: proxy answer <none>
2025-10-31T10:10:38.409175Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.409206Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.409222Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic-mirrored-from-dc2, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
CallPersQueueGRPC request to localhost:3655
MetaRequest {
  CmdCreateTopic {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc2"
    NumPartitions: 1
    Config {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      Consumers {
        Name: "user"
        ReadFromTimestampsMs: 0
        FormatVersion: 0
        Version: 0
      }
    }
  }
}
CallPersQueueGRPC response:
Status: 129
ProxyErrorCode: 53
SchemeStatus: 1
FlatTxId {
  TxId: 281474976710674
  SchemeShardTabletId: 72057594046644480
  PathId: 14
}
ErrorCode: OK
AddTopic: rt3.dc1--test-topic-mirrored-from-dc2
===Run query:``DECLARE $version as Int64; DECLARE $path AS Utf8; DECLARE $cluster as Utf8; UPSERT INTO `/Root/PQ/Config/V2/Topics` (path, dc) VALUES ($path, $cluster); UPSERT INTO `/Root/PQ/Config/V2/Versions` (name, version) VALUES ("Topics", $version);`` with topic = test-topic-mirrored-from-dc2, dc = dc1
2025-10-31T10:10:38.476964Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037895][] pipe [3:7567326239128211175:3407] connected; active server actors: 1
2025-10-31T10:10:38.477176Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1519: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] updating configuration. Deleted partitions []. Added partitions [0]
2025-10-31T10:10:38.478217Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer.cpp:971: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] Discovered subdomain [OwnerId: 72057594046644480, LocalPathId: 1] state, outOfSpace = 0 at RB 72075186224037895
2025-10-31T10:10:38.480094Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer.cpp:140: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] BALANCER INIT DONE for rt3.dc1--test-topic-mirrored-from-dc2: (0, 72075186224037894) 
2025-10-31T10:10:38.482311Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2960: [PQ: 72075186224037894] Handle TEvInterconnect::TEvNodeInfo
2025-10-31T10:10:38.485737Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2992: [PQ: 72075186224037894] Transactions request. From tx_00000000000000000000, To tx_18446744073709551615
2025-10-31T10:10:38.486021Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:787: [PQ: 72075186224037894] no config, start with empty partitions and default config
2025-10-31T10:10:38.486050Z node 4 :PERSQUEUE NOTICE: pq_impl.cpp:913: [PQ: 72075186224037894] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-10-31T10:10:38.486090Z node 4 :PERSQUEUE INFO: pq_impl.cpp:612: [PQ: 72075186224037894] doesn't have tx writes info
2025-10-31T10:10:38.487658Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037894] server connected, pipe [3:7567326239128211174:3406], now have 1 active actors on pipe
2025-10-31T10:10:38.487690Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037894] server connected, pipe [3:7567326239128211201:2465], now have 1 active actors on pipe
2025-10-31T10:10:38.488091Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.489306Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer.cpp:404: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] TEvClientConnected TabletId 72075186224037894, NodeId 4, Generation 1
2025-10-31T10:10:38.497486Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.510152Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.510185Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.510199Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.510769Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037894] server connected, pipe [3:7567326239128211232:3451], now have 1 active actors on pipe
2025-10-31T10:10:38.518111Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.527633Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.528688Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:81: [rt3.dc1--test-topic-mirrored-from-dc2:0:Initializer] Start initializing step TInitConfigStep
2025-10-31T10:10:38.528747Z node 4 :PERSQUEUE DEBUG: partition.cpp:1295: [72075186224037894][Partition][0][StateInit] HandleOnInit TEvPQ::TEvProposePartitionConfig
2025-10-31T10:10:38.529091Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:81: [rt3.dc1--test-topic-mirrored-from-dc2:0:Initializer] Start initializing step TInitInternalFieldsStep
2025-10-31T10:10:38.529347Z node 4 :PERSQUEUE INFO: partition_init.cpp:1101: [72075186224037894][Partition][0][StateInit] bootstrapping 0 [4:7567326236120278927:2397]
2025-10-31T10:10:38.531857Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:81: [rt3.dc1--test-topic-mirrored-from-dc2:0:Initializer] Start initializing step TInitFieldsStep
2025-10-31T10:10:38.531880Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:61: [rt3.dc1--test-topic-mirrored-from-dc2:0:Initializer] Initializing completed.
2025-10-31T10:10:38.531903Z node 4 :PERSQUEUE INFO: partition.cpp:707: [72075186224037894][Partition][0][StateInit] init complete for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 generation 1 [4:7567326236120278927:2397]
2025-10-31T10:10:38.531932Z node 4 :PERSQUEUE DEBUG: partition.cpp:721: [72075186224037894][Partition][0][StateInit] SYNC INIT topic rt3.dc1--test-topic-mirrored-from-dc2 partitition 0 so 0 endOffset 0 Head Offset 0 PartNo 0 PackedSize 0 count 0 nextOffset 0 batches 0
2025-10-31T10:10:38.531964Z node 4 :PERSQUEUE DEBUG: partition_mlp.cpp:83: [72075186224037894][Partition][0][StateIdle] Initializing MLP Consumers: 0
2025-10-31T10:10:38.536226Z node 4 :PERSQUEUE DEBUG: partition.cpp:4256: [72075186224037894][Partition][0][StateIdle] Process pending events. Count 1
2025-10-31T10:10:38.536268Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.536348Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.536359Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037894][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:38.536376Z node 4 :PERSQUEUE DEBUG: partition.cpp:2301: [72075186224037894][Partition][0][StateIdle] Still pending for come commits
2025-10-31T10:10:38.536388Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.536398Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.536425Z node 4 :PERSQUEUE DEBUG: partition.cpp:2301: [72075186224037894][Partition][0][StateIdle] Still pending for come commits
2025-10-31T10:10:38.536458Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:926: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-10-31T10:10:38.536534Z node 4 :PERSQUEUE DEBUG: partition_mlp.cpp:60: [72075186224037894][Partition][0][StateIdle] Process MLP pending events. Count 0
2025-10-31T10:10:38.536853Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.536979Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:183: [72075186224037894][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:38.544907Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.545065Z node 4 :PERSQUEUE DEBUG: partition.cpp:1399: [72075186224037894][Partition][0][StateIdle] Handle TEvPQ::TEvTxCommit Step 1761905438552, TxId 281474976710674
2025-10-31T10:10:38.545109Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.545123Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.545242Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.545448Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037894][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:38.549629Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037894][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:38.549774Z node 4 :PERSQUEUE DEBUG: partition_mlp.cpp:83: [72075186224037894][Partition][0][StateIdle] Initializing MLP Consumers: 0
2025-10-31T10:10:38.549878Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037894][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:38.549899Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.549917Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.549929Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.549960Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:183: [72075186224037894][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:38.550291Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:398: [PQ: 72075186224037894] Apply new config PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc2" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Version: 0 Generation: 0 } MonitoringProjectId: ""
2025-10-31T10:10:38.550429Z node 4 :PERSQUEUE NOTICE: pq_impl.cpp:913: [PQ: 72075186224037894] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-10-31T10:10:38.550681Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.564456Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.564565Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.572129Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.607136Z node 3 :KQP_EXECUTER ERROR: kqp_planner.cpp:123: TxId: 281474976710675. Ctx: { TraceId: 01k8wvyvr617t6jraxwxvnv1ys, Database: , SessionId: ydb://session/3?node_id=3&id=N2YwMTQ4YmMtMzJkYjQ4MTktZTBkMGUzMzctMTMxYzNhYTU=, PoolId: default, DatabaseId: /Root}. Database not set, use /Root
2025-10-31T10:10:38.611276Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.611309Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.611325Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.635575Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.635609Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.635624Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
===Query complete
TClient::Ls request: /Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2
TClient::Ls response: Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic-mirrored-from-dc2"
    PathId: 14
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976710674
    CreateStep: 1761905438552
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic-mirrored-from-dc2"
    PathId: 14
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize...
(TRUNCATED)
GetTopicVersionFromPath:  record Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic-mirrored-from-dc2"
    PathId: 14
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976710674
    CreateStep: 1761905438552
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic-mirrored-from-dc2"
    PathId: 14
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      YdbDatabasePath: "/Root"
      Consumers {
        Name: "user"
        ReadFromTimestampsMs: 0
        FormatVersion: 0
        Version: 0
      }
    }
    Partitions {
      PartitionId: 0
      TabletId: 72075186224037894
      Status: Active
    }
    AlterVersion: 1
    BalancerTabletID: 72075186224037895
    NextPartitionId: 1
  }
  DomainDescription {
    SchemeShardId_Depricated: 72057594046644480
    PathId_Depricated: 1
    ProcessingParams {
      Version: 1
      PlanResolution: 7
      Coordinators: 72057594046316545
      TimeCastBucketsPerMediator: 2
      Mediators: 72057594046382081
    }
    DomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    PathsInside: 13
    PathsLimit: 10000
    ShardsInside: 8
    ShardsLimit: 200000
    ResourcesDomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    DiskSpaceUsage {
      Tables {
        TotalSize: 0
        DataSize: 0
        IndexSize: 0
      }
      Topics {
        ReserveSize: 0
        AccountSize: 0
        DataSize: 0
        UsedReserveSize: 0
      }
    }
    PQPartitionsInside: 2
    PQPartitionsLimit: 1000000
    SchemeLimits {
      MaxDepth: 32
      MaxPaths: 10000
      MaxChildrenInDir: 100000
      MaxAclBytesSize: 10240
      MaxTableColumns: 200
      MaxTableColumnNameLength: 255
      MaxTableKeyColumns: 30
      MaxTableIndices: 20
      MaxShards: 200000
      MaxShardsInPath: 35000
      MaxConsistentCopyTargets: 10000
      MaxPathElementLength: 255
      ExtraPathSymbolsAllowed: "!\"#$%&\'()*+,-.:;<=>?@[\\]^_`{|}~"
      MaxPQPartitions: 1000000
      MaxTableCdcStreams: 5
      MaxExports: 10
      MaxImports: 10
      MaxColumnTableColumns: 10000
    }
  }
}
Path: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2"
 name rt3.dc1--test-topic-mirrored-from-dc2 version1
CallPersQueueGRPC request to localhost:3655
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc2"
  }
}
2025-10-31T10:10:38.651645Z node 3 :PERSQUEUE INFO: msgbus_server_persqueue.cpp:1524: proxy answer <none>
CallPersQueueGRPC response:
Status: 1
ErrorCode: OK
MetaResponse {
  CmdGetTopicMetadataResult {
    TopicInfo {
      Topic: "rt3.dc1--test-topic-mirrored-from-dc2"
      NumPartitions: 1
      Config {
        PartitionConfig {
          LifetimeSeconds: 86400
          LowWatermark: 8388608
          SourceIdLifetimeSeconds: 86400
          WriteSpeedInBytesPerSecond: 20000000
          BurstSize: 20000000
          SourceIdMaxCounts: 6000000
        }
        Version: 1
        LocalDC: true
        Codecs {
          Ids: 0
          Ids: 1
          Ids: 2
          Codecs: "raw"
          Codecs: "gzip"
          Codecs: "lzop"
        }
        TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2"
        YdbDatabasePath: "/Root"
        Consumers {
          Name: "user"
          ReadFromTimestampsMs: 0
          FormatVersion: 0
          Version: 0
        }
      }
      ErrorCode: OK
    }
  }
}
=== Topic created, have version: 1
PQ Client: create topic: rt3.dc1--test-topic-mirrored-from-dc3 with 1 partitions
CallPersQueueGRPC request to localhost:3655
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc3"
  }
}
2025-10-31T10:10:38.656014Z node 3 :PERSQUEUE INFO: msgbus_server_persqueue.cpp:1524: proxy answer <none>
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic-mirrored-from-dc3, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
CallPersQueueGRPC request to localhost:3655
MetaRequest {
  CmdCreateTopic {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc3"
    NumPartitions: 1
    Config {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      Consumers {
        Name: "user"
        ReadFromTimestampsMs: 0
        FormatVersion: 0
        Version: 0
      }
    }
  }
}
CallPersQueueGRPC response:
Status: 129
ProxyErrorCode: 53
SchemeStatus: 1
FlatTxId {
  TxId: 281474976710676
  SchemeShardTabletId: 72057594046644480
  PathId: 15
}
ErrorCode: OK
AddTopic: rt3.dc1--test-topic-mirrored-from-dc3
===Run query:``DECLARE $version as Int64; DECLARE $path AS Utf8; DECLARE $cluster as Utf8; UPSERT INTO `/Root/PQ/Config/V2/Topics` (path, dc) VALUES ($path, $cluster); UPSERT INTO `/Root/PQ/Config/V2/Versions` (name, version) VALUES ("Topics", $version);`` with topic = test-topic-mirrored-from-dc3, dc = dc1
2025-10-31T10:10:38.713185Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.713213Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.713225Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.718853Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2960: [PQ: 72075186224037896] Handle TEvInterconnect::TEvNodeInfo
2025-10-31T10:10:38.726466Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2992: [PQ: 72075186224037896] Transactions request. From tx_00000000000000000000, To tx_18446744073709551615
2025-10-31T10:10:38.726785Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:787: [PQ: 72075186224037896] no config, start with empty partitions and default config
2025-10-31T10:10:38.726811Z node 4 :PERSQUEUE NOTICE: pq_impl.cpp:913: [PQ: 72075186224037896] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-10-31T10:10:38.726846Z node 4 :PERSQUEUE INFO: pq_impl.cpp:612: [PQ: 72075186224037896] doesn't have tx writes info
2025-10-31T10:10:38.742106Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.742138Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.742153Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.785441Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037896] server connected, pipe [3:7567326239128211402:3562], now have 1 active actors on pipe
2025-10-31T10:10:38.786597Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.798180Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.813896Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.813927Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.813940Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.842217Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.842244Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.842257Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.853753Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037897][] pipe [3:7567326239128211403:3563] connected; active server actors: 1
2025-10-31T10:10:38.853923Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1519: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] updating configuration. Deleted partitions []. Added partitions [0]
2025-10-31T10:10:38.861686Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer.cpp:971: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] Discovered subdomain [OwnerId: 72057594046644480, LocalPathId: 1] state, outOfSpace = 0 at RB 72075186224037897
2025-10-31T10:10:38.861791Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer.cpp:140: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] BALANCER INIT DONE for rt3.dc1--test-topic-mirrored-from-dc3: (0, 72075186224037896) 
2025-10-31T10:10:38.864429Z node 3 :KQP_EXECUTER ERROR: kqp_planner.cpp:123: TxId: 281474976710677. Ctx: { TraceId: 01k8wvyvzk6pg3hxdx23mqgn2g, Database: , SessionId: ydb://session/3?node_id=3&id=MmQzMTYwNzQtYzIxNDJhOTgtNGE3MGFiMmYtYTM2Y2Y4NjU=, PoolId: default, DatabaseId: /Root}. Database not set, use /Root
2025-10-31T10:10:38.865259Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037896] server connected, pipe [3:7567326239128211416:2485], now have 1 active actors on pipe
2025-10-31T10:10:38.866332Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer.cpp:404: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] TEvClientConnected TabletId 72075186224037896, NodeId 4, Generation 1
2025-10-31T10:10:38.878763Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037896] server connected, pipe [3:7567326239128211444:3590], now have 1 active actors on pipe
2025-10-31T10:10:38.880427Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.890028Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.891179Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:81: [rt3.dc1--test-topic-mirrored-from-dc3:0:Initializer] Start initializing step TInitConfigStep
2025-10-31T10:10:38.891232Z node 4 :PERSQUEUE DEBUG: partition.cpp:1295: [72075186224037896][Partition][0][StateInit] HandleOnInit TEvPQ::TEvProposePartitionConfig
2025-10-31T10:10:38.891518Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:81: [rt3.dc1--test-topic-mirrored-from-dc3:0:Initializer] Start initializing step TInitInternalFieldsStep
2025-10-31T10:10:38.891707Z node 4 :PERSQUEUE INFO: partition_init.cpp:1101: [72075186224037896][Partition][0][StateInit] bootstrapping 0 [4:7567326236120279016:2404]
2025-10-31T10:10:38.894291Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:81: [rt3.dc1--test-topic-mirrored-from-dc3:0:Initializer] Start initializing step TInitFieldsStep
2025-10-31T10:10:38.894322Z node 4 :PERSQUEUE DEBUG: partition_init.cpp:61: [rt3.dc1--test-topic-mirrored-from-dc3:0:Initializer] Initializing completed.
2025-10-31T10:10:38.894345Z node 4 :PERSQUEUE INFO: partition.cpp:707: [72075186224037896][Partition][0][StateInit] init complete for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 generation 1 [4:7567326236120279016:2404]
2025-10-31T10:10:38.894369Z node 4 :PERSQUEUE DEBUG: partition.cpp:721: [72075186224037896][Partition][0][StateInit] SYNC INIT topic rt3.dc1--test-topic-mirrored-from-dc3 partitition 0 so 0 endOffset 0 Head Offset 0 PartNo 0 PackedSize 0 count 0 nextOffset 0 batches 0
2025-10-31T10:10:38.894399Z node 4 :PERSQUEUE DEBUG: partition_mlp.cpp:83: [72075186224037896][Partition][0][StateIdle] Initializing MLP Consumers: 0
2025-10-31T10:10:38.895683Z node 4 :PERSQUEUE DEBUG: partition.cpp:4256: [72075186224037896][Partition][0][StateIdle] Process pending events. Count 1
2025-10-31T10:10:38.895713Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.895792Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.895805Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037896][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:38.895825Z node 4 :PERSQUEUE DEBUG: partition.cpp:2301: [72075186224037896][Partition][0][StateIdle] Still pending for come commits
2025-10-31T10:10:38.895837Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.895848Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.895860Z node 4 :PERSQUEUE DEBUG: partition.cpp:2301: [72075186224037896][Partition][0][StateIdle] Still pending for come commits
2025-10-31T10:10:38.895897Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:926: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-10-31T10:10:38.895932Z node 4 :PERSQUEUE DEBUG: partition_mlp.cpp:60: [72075186224037896][Partition][0][StateIdle] Process MLP pending events. Count 0
2025-10-31T10:10:38.896214Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.896330Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:183: [72075186224037896][Partition][0][StateIdle] No data for blobs compaction
===Query complete
TClient::Ls request: /Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3
TClient::Ls response: Status: 128
StatusCode: PATH_NOT_EXIST
Issues {
  message: "Path not exist"
  issue_code: 200200
  severity: 1
}
SchemeStatus: 2
ErrorReason: "Path not found"
GetTopicVersionFromPath:  record Status: 128
StatusCode: PATH_NOT_EXIST
Issues {
  message: "Path not exist"
  issue_code: 200200
  severity: 1
}
SchemeStatus: 2
ErrorReason: "Path not found"
 name rt3.dc1--test-topic-mirrored-from-dc3 version0
2025-10-31T10:10:38.903977Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.904121Z node 4 :PERSQUEUE DEBUG: partition.cpp:1399: [72075186224037896][Partition][0][StateIdle] Handle TEvPQ::TEvTxCommit Step 1761905438923, TxId 281474976710676
2025-10-31T10:10:38.904142Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.904155Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.904259Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.904449Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037896][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:38.908085Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037896][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:38.908223Z node 4 :PERSQUEUE DEBUG: partition_mlp.cpp:83: [72075186224037896][Partition][0][StateIdle] Initializing MLP Consumers: 0
2025-10-31T10:10:38.908327Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037896][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:38.908350Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.908364Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.908374Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.910515Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:183: [72075186224037896][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:38.910938Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:398: [PQ: 72075186224037896] Apply new config PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc3" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Version: 0 Generation: 0 } MonitoringProjectId: ""
2025-10-31T10:10:38.911051Z node 4 :PERSQUEUE NOTICE: pq_impl.cpp:913: [PQ: 72075186224037896] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-10-31T10:10:38.911293Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.918515Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.918550Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.918566Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.926659Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.926765Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:3570: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-10-31T10:10:38.929922Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1045: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-10-31T10:10:38.945060Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.945091Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.945119Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:38.997198Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:38.997228Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:38.997243Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:39.021684Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:39.021730Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:39.021745Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:39.056857Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:39.056897Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:39.056911Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:39.098037Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:39.098070Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:39.098086Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:39.123671Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:39.123701Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:39.123716Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:39.156148Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:39.156175Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:39.156188Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:39.201212Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:39.201256Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:39.201270Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:39.227638Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:39.227672Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:39.227687Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
TClient::Ls request: /Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3
TClient::Ls response: Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic-mirrored-from-dc3"
    PathId: 15
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976710676
    CreateStep: 1761905438923
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic-mirrored-from-dc3"
    PathId: 15
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize...
(TRUNCATED)
GetTopicVersionFromPath:  record Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic-mirrored-from-dc3"
    PathId: 15
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976710676
    CreateStep: 1761905438923
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic-mirrored-from-dc3"
    PathId: 15
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      YdbDatabasePath: "/Root"
      Consumers {
        Name: "user"
        ReadFromTimestampsMs: 0
        FormatVersion: 0
        Version: 0
      }
    }
    Partitions {
      PartitionId: 0
      TabletId: 72075186224037896
      Status: Active
    }
    AlterVersion: 1
    BalancerTabletID: 72075186224037897
    NextPartitionId: 1
  }
  DomainDescription {
    SchemeShardId_Depricated: 72057594046644480
    PathId_Depricated: 1
    ProcessingParams {
      Version: 1
      PlanResolution: 7
      Coordinators: 72057594046316545
      TimeCastBucketsPerMediator: 2
      Mediators: 72057594046382081
    }
    DomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    PathsInside: 14
    PathsLimit: 10000
    ShardsInside: 10
    ShardsLimit: 200000
    ResourcesDomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    DiskSpaceUsage {
      Tables {
        TotalSize: 0
        DataSize: 0
        IndexSize: 0
      }
      Topics {
        ReserveSize: 0
        AccountSize: 0
        DataSize: 0
        UsedReserveSize: 0
      }
    }
    PQPartitionsInside: 3
    PQPartitionsLimit: 1000000
    SchemeLimits {
      MaxDepth: 32
      MaxPaths: 10000
      MaxChildrenInDir: 100000
      MaxAclBytesSize: 10240
      MaxTableColumns: 200
      MaxTableColumnNameLength: 255
      MaxTableKeyColumns: 30
      MaxTableIndices: 20
      MaxShards: 200000
      MaxShardsInPath: 35000
      MaxConsistentCopyTargets: 10000
      MaxPathElementLength: 255
      ExtraPathSymbolsAllowed: "!\"#$%&\'()*+,-.:;<=>?@[\\]^_`{|}~"
      MaxPQPartitions: 1000000
      MaxTableCdcStreams: 5
      MaxExports: 10
      MaxImports: 10
      MaxColumnTableColumns: 10000
    }
  }
}
Path: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3"
 name rt3.dc1--test-topic-mirrored-from-dc3 version1
CallPersQueueGRPC request to localhost:3655
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc3"
  }
}
2025-10-31T10:10:39.414147Z node 3 :PERSQUEUE INFO: msgbus_server_persqueue.cpp:1524: proxy answer <none>
CallPersQueueGRPC response:
Status: 1
ErrorCode: OK
MetaResponse {
  CmdGetTopicMetadataResult {
    TopicInfo {
      Topic: "rt3.dc1--test-topic-mirrored-from-dc3"
      NumPartitions: 1
      Config {
        PartitionConfig {
          LifetimeSeconds: 86400
          LowWatermark: 8388608
          SourceIdLifetimeSeconds: 86400
          WriteSpeedInBytesPerSecond: 20000000
          BurstSize: 20000000
          SourceIdMaxCounts: 6000000
        }
        Version: 1
        LocalDC: true
        Codecs {
          Ids: 0
          Ids: 1
          Ids: 2
          Codecs: "raw"
          Codecs: "gzip"
          Codecs: "lzop"
        }
        TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3"
        YdbDatabasePath: "/Root"
        Consumers {
          Name: "user"
          ReadFromTimestampsMs: 0
          FormatVersion: 0
          Version: 0
        }
      }
      ErrorCode: OK
    }
  }
}
=== Topic created, have version: 1
>>> Session was created
>>> Ready to answer: ok
2025-10-31T10:10:39.497157Z :INFO: [/Root] [] [6853a82c-28928105-f39bccb1-583f0885] Open read subsessions to databases: { name: dc1, endpoint: localhost:3655, path: /Root }
2025-10-31T10:10:39.518086Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: try to update token
2025-10-31T10:10:39.518296Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session: Do CDS request
2025-10-31T10:10:39.518336Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Start write session. Will connect to endpoint: localhost:3655
2025-10-31T10:10:39.520187Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Starting read session
2025-10-31T10:10:39.520240Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Starting single session
2025-10-31T10:10:39.521161Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In Reconnect, ReadSizeBudget = 8388608, ReadSizeServerDelta = 0
2025-10-31T10:10:39.521293Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] New values: ReadSizeBudget = 8388608, ReadSizeServerDelta = 0
2025-10-31T10:10:39.522149Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Reconnecting session to cluster  in 0.000000s
2025-10-31T10:10:39.527991Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Successfully connected. Initializing session
2025-10-31T10:10:39.530281Z node 3 :PQ_READ_PROXY DEBUG: grpc_pq_read.h:109: new grpc connection
2025-10-31T10:10:39.530311Z node 3 :PQ_READ_PROXY DEBUG: grpc_pq_read.h:131: new session created cookie 1
2025-10-31T10:10:39.530990Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer  session  grpc read done: success# 1, data# { init_request { topics_read_settings { path: "test-topic" } topics_read_settings { path: "test-topic-mirrored-from-dc2" } topics_read_settings { path: "test-topic-mirrored-from-dc3" } consumer: "shared/user" } }
2025-10-31T10:10:39.531246Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:929: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 read init: from# ipv6:[::1]:33546, request# { init_request { topics_read_settings { path: "test-topic" } topics_read_settings { path: "test-topic-mirrored-from-dc2" } topics_read_settings { path: "test-topic-mirrored-from-dc3" } consumer: "shared/user" } }
2025-10-31T10:10:39.531627Z node 3 :PQ_READ_PROXY DEBUG: read_init_auth_actor.cpp:41: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 auth for : user
2025-10-31T10:10:39.532361Z node 3 :PQ_READ_PROXY DEBUG: read_init_auth_actor.cpp:131: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Handle describe topics response
2025-10-31T10:10:39.532568Z node 3 :PQ_READ_PROXY DEBUG: read_init_auth_actor.cpp:68: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 auth is DEAD
2025-10-31T10:10:39.532648Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:1046: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 auth ok: topics# 3, initDone# 0
2025-10-31T10:10:39.534603Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Got InitResponse. ReadSessionId: shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:39.534648Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 8388608, ReadSizeServerDelta = 0
2025-10-31T10:10:39.534798Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: send init request: init_request { topic: "test-topic" message_group_id: "src_id" }
2025-10-31T10:10:39.534849Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-10-31T10:10:39.533791Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:1217: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 register session: topic# rt3.dc1--test-topic-mirrored-from-dc3
2025-10-31T10:10:39.533827Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:1217: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 register session: topic# rt3.dc1--test-topic
2025-10-31T10:10:39.533840Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:1217: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 register session: topic# rt3.dc1--test-topic-mirrored-from-dc2
2025-10-31T10:10:39.533994Z node 3 :PQ_WRITE_PROXY DEBUG: grpc_pq_write.h:107: new grpc connection
2025-10-31T10:10:39.534007Z node 3 :PQ_WRITE_PROXY DEBUG: grpc_pq_write.h:141: new session created cookie 2
2025-10-31T10:10:39.534137Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7567326243423178831:2513] connected; active server actors: 1
2025-10-31T10:10:39.534166Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: read_balancer__balancing.cpp:1706: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer "user" register session for pipe [3:7567326243423178831:2513] session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:39.534203Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:639: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user register readable partition 0
2025-10-31T10:10:39.534260Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:669: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user family created family=1 (Status=Free, Partitions=[0])
2025-10-31T10:10:39.534294Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:871: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user register reading session ReadingSession "shared/user_3_1_14607229299462673975_v1" (Sender=[3:7567326243423178828:2513], Pipe=[3:7567326243423178831:2513], Partitions=[], ActiveFamilyCount=0)
2025-10-31T10:10:39.534315Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1186: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user rebalancing was scheduled
2025-10-31T10:10:39.534356Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1258: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user balancing. Sessions=1, Families=1, UnradableFamilies=1 [1 (0), ], RequireBalancing=0 []
2025-10-31T10:10:39.534400Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1305: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user balancing family=1 (Status=Free, Partitions=[0]) for ReadingSession "shared/user_3_1_14607229299462673975_v1" (Sender=[3:7567326243423178828:2513], Pipe=[3:7567326243423178831:2513], Partitions=[], ActiveFamilyCount=0)
2025-10-31T10:10:39.534454Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:549: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user family 1 status Active partitions [0] session "shared/user_3_1_14607229299462673975_v1" sender [3:7567326243423178828:2513] lock partition 0 for ReadingSession "shared/user_3_1_14607229299462673975_v1" (Sender=[3:7567326243423178828:2513], Pipe=[3:7567326243423178831:2513], Partitions=[], ActiveFamilyCount=1) generation 1 step 1
2025-10-31T10:10:39.534494Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1325: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user start rebalancing. familyCount=1, sessionCount=1, desiredFamilyCount=1, allowPlusOne=0
2025-10-31T10:10:39.534521Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1402: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user balancing duration: 0.000143s
2025-10-31T10:10:39.534548Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326243423178832:2513] connected; active server actors: 1
2025-10-31T10:10:39.534584Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: read_balancer__balancing.cpp:1706: [72075186224037893][rt3.dc1--test-topic] consumer "user" register session for pipe [3:7567326243423178832:2513] session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:39.534590Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7567326243423178833:2513] connected; active server actors: 1
2025-10-31T10:10:39.534607Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:639: [72075186224037893][rt3.dc1--test-topic] consumer user register readable partition 0
2025-10-31T10:10:39.534630Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: read_balancer__balancing.cpp:1706: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer "user" register session for pipe [3:7567326243423178833:2513] session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:39.534632Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:669: [72075186224037893][rt3.dc1--test-topic] consumer user family created family=1 (Status=Free, Partitions=[0])
2025-10-31T10:10:39.534656Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:871: [72075186224037893][rt3.dc1--test-topic] consumer user register reading session ReadingSession "shared/user_3_1_14607229299462673975_v1" (Sender=[3:7567326243423178828:2513], Pipe=[3:7567326243423178832:2513], Partitions=[], ActiveFamilyCount=0)
2025-10-31T10:10:39.534667Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:639: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user register readable partition 0
2025-10-31T10:10:39.534670Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1186: [72075186224037893][rt3.dc1--test-topic] consumer user rebalancing was scheduled
2025-10-31T10:10:39.534699Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1258: [72075186224037893][rt3.dc1--test-topic] consumer user balancing. Sessions=1, Families=1, UnradableFamilies=1 [1 (0), ], RequireBalancing=0 []
2025-10-31T10:10:39.534714Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:669: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user family created family=1 (Status=Free, Partitions=[0])
2025-10-31T10:10:39.534731Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1305: [72075186224037893][rt3.dc1--test-topic] consumer user balancing family=1 (Status=Free, Partitions=[0]) for ReadingSession "shared/user_3_1_14607229299462673975_v1" (Sender=[3:7567326243423178828:2513], Pipe=[3:7567326243423178832:2513], Partitions=[], ActiveFamilyCount=0)
2025-10-31T10:10:39.534746Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:871: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user register reading session ReadingSession "shared/user_3_1_14607229299462673975_v1" (Sender=[3:7567326243423178828:2513], Pipe=[3:7567326243423178833:2513], Partitions=[], ActiveFamilyCount=0)
2025-10-31T10:10:39.534766Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1186: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user rebalancing was scheduled
2025-10-31T10:10:39.534769Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:549: [72075186224037893][rt3.dc1--test-topic] consumer user family 1 status Active partitions [0] session "shared/user_3_1_14607229299462673975_v1" sender [3:7567326243423178828:2513] lock partition 0 for ReadingSession "shared/user_3_1_14607229299462673975_v1" (Sender=[3:7567326243423178828:2513], Pipe=[3:7567326243423178832:2513], Partitions=[], ActiveFamilyCount=1) generation 1 step 1
2025-10-31T10:10:39.534796Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1325: [72075186224037893][rt3.dc1--test-topic] consumer user start rebalancing. familyCount=1, sessionCount=1, desiredFamilyCount=1, allowPlusOne=0
2025-10-31T10:10:39.534801Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1258: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user balancing. Sessions=1, Families=1, UnradableFamilies=1 [1 (0), ], RequireBalancing=0 []
2025-10-31T10:10:39.534813Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1402: [72075186224037893][rt3.dc1--test-topic] consumer user balancing duration: 0.000099s
2025-10-31T10:10:39.534840Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1305: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user balancing family=1 (Status=Free, Partitions=[0]) for ReadingSession "shared/user_3_1_14607229299462673975_v1" (Sender=[3:7567326243423178828:2513], Pipe=[3:7567326243423178833:2513], Partitions=[], ActiveFamilyCount=0)
2025-10-31T10:10:39.534887Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:549: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user family 1 status Active partitions [0] session "shared/user_3_1_14607229299462673975_v1" sender [3:7567326243423178828:2513] lock partition 0 for ReadingSession "shared/user_3_1_14607229299462673975_v1" (Sender=[3:7567326243423178828:2513], Pipe=[3:7567326243423178833:2513], Partitions=[], ActiveFamilyCount=1) generation 1 step 1
2025-10-31T10:10:39.534927Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1325: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user start rebalancing. familyCount=1, sessionCount=1, desiredFamilyCount=1, allowPlusOne=0
2025-10-31T10:10:39.534957Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1402: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user balancing duration: 0.000131s
2025-10-31T10:10:39.535587Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:1335: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 assign: record# { Partition: 0 TabletId: 72075186224037896 Topic: "rt3.dc1--test-topic-mirrored-from-dc3" Generation: 1 Step: 1 Session: "shared/user_3_1_14607229299462673975_v1" ClientId: "user" PipeClient { RawX1: 7567326243423178831 RawX2: 4503612512274897 } Path: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3" }
2025-10-31T10:10:39.536137Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:1335: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 assign: record# { Partition: 0 TabletId: 72075186224037892 Topic: "rt3.dc1--test-topic" Generation: 1 Step: 1 Session: "shared/user_3_1_14607229299462673975_v1" ClientId: "user" PipeClient { RawX1: 7567326243423178832 RawX2: 4503612512274897 } Path: "/Root/PQ/rt3.dc1--test-topic" }
2025-10-31T10:10:39.536668Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 2 sessionId:  grpc read done: success: 1 data: init_request { topic: "test-topic" message_group_id: "src_id" }
2025-10-31T10:10:39.536692Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:1335: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 assign: record# { Partition: 0 TabletId: 72075186224037894 Topic: "rt3.dc1--test-topic-mirrored-from-dc2" Generation: 1 Step: 1 Session: "shared/user_3_1_14607229299462673975_v1" ClientId: "user" PipeClient { RawX1: 7567326243423178833 RawX2: 4503612512274897 } Path: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2" }
2025-10-31T10:10:39.536740Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { read_request { bytes_size: 8388608 } }
2025-10-31T10:10:39.536797Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:447: session request cookie: 2 topic: "test-topic" message_group_id: "src_id" from ipv6:[::1]:33546
2025-10-31T10:10:39.536813Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:1538: write session:  cookie=2 sessionId= userAgent="pqv1 server" ip=ipv6:[::1]:33546 proto=v1  topic=test-topic durationSec=0
2025-10-31T10:10:39.536824Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1839: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got read request: guid# 4f6b4d49-708b3e8d-e858c47a-46e2252b
2025-10-31T10:10:39.536827Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:571: init check schema
2025-10-31T10:10:39.536902Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:1143: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 INITING TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1)
2025-10-31T10:10:39.536903Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:1143: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 INITING TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2)
2025-10-31T10:10:39.536970Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:1143: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 INITING TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3)
2025-10-31T10:10:39.540101Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:632: session v1 cookie: 2 sessionId:  describe result for acl check
2025-10-31T10:10:39.540212Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:62: TTableHelper SelectQuery: --!syntax_v1
DECLARE $Hash AS Uint32; DECLARE $Topic AS Utf8; DECLARE $SourceId AS Utf8;
SELECT Partition, CreateTime, AccessTime, SeqNo FROM `/Root/PQ/SourceIdMeta2` WHERE Hash == $Hash AND Topic == $Topic AND SourceId == $SourceId;
2025-10-31T10:10:39.540239Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:63: TTableHelper UpdateQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;DECLARE $SeqNo AS Uint64;
UPSERT INTO `/Root/PQ/SourceIdMeta2` (Hash, Topic, SourceId, CreateTime, AccessTime, Partition, SeqNo) VALUES ($Hash, $Topic, $SourceId, $CreateTime, $AccessTime, $Partition, $SeqNo);
2025-10-31T10:10:39.540249Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:64: TTableHelper UpdateAccessTimeQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;
UPDATE `/Root/PQ/SourceIdMeta2` SET AccessTime = $AccessTime WHERE Hash = $Hash AND Topic = $Topic AND SourceId = $SourceId AND Partition = $Partition;
2025-10-31T10:10:39.540265Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:111: TPartitionChooser [3:7567326243423178845:2516] (SourceId=src_id, PreferedPartition=(NULL)) StartKqpSession
2025-10-31T10:10:39.540408Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:983: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) pipe restart attempt 0 pipe creation result: OK TabletId: 72075186224037892 Generation: 1, pipe: [3:7567326243423178842:2518]
2025-10-31T10:10:39.540464Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:983: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) pipe restart attempt 0 pipe creation result: OK TabletId: 72075186224037894 Generation: 1, pipe: [3:7567326243423178843:2519]
2025-10-31T10:10:39.541566Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:983: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) pipe restart attempt 0 pipe creation result: OK TabletId: 72075186224037896 Generation: 1, pipe: [3:7567326243423178841:2517]
2025-10-31T10:10:39.543378Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:142: TPartitionChooser [3:7567326243423178845:2516] (SourceId=src_id, PreferedPartition=(NULL)) Select from the table
2025-10-31T10:10:39.552416Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__old_chooser_actor.h:67: TPartitionChooser [3:7567326243423178845:2516] (SourceId=src_id, PreferedPartition=(NULL)) RequestPQRB
2025-10-31T10:10:39.552863Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326243423178865:2516] connected; active server actors: 1
2025-10-31T10:10:39.552908Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__old_chooser_actor.h:80: TPartitionChooser [3:7567326243423178845:2516] (SourceId=src_id, PreferedPartition=(NULL)) Received partition 0 from PQRB for SourceId=src_id
2025-10-31T10:10:39.552932Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:174: TPartitionChooser [3:7567326243423178845:2516] (SourceId=src_id, PreferedPartition=(NULL)) Update the table
2025-10-31T10:10:39.553345Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1659: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326243423178865:2516] disconnected.
2025-10-31T10:10:39.553367Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1670: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326243423178865:2516] disconnected; active server actors: 1
2025-10-31T10:10:39.553384Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1695: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326243423178865:2516] disconnected no session
2025-10-31T10:10:39.565057Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:183: TPartitionChooser [3:7567326243423178845:2516] (SourceId=src_id, PreferedPartition=(NULL)) HandleUpdate PartitionPersisted=0 Status=SUCCESS
2025-10-31T10:10:39.565111Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:305: TPartitionChooser [3:7567326243423178845:2516] (SourceId=src_id, PreferedPartition=(NULL)) ReplyResult: Partition=0, SeqNo=(NULL)
2025-10-31T10:10:39.565127Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:268: TPartitionChooser [3:7567326243423178845:2516] (SourceId=src_id, PreferedPartition=(NULL)) Start idle
2025-10-31T10:10:39.565154Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:694: ProceedPartition. session cookie: 2 sessionId:  partition: 0 expectedGeneration: (NULL)
2025-10-31T10:10:39.567004Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:822: TPartitionWriter 72075186224037892 (partition=0) TEvClientConnected Status OK, TabletId: 72075186224037892, NodeId 4, Generation: 1
2025-10-31T10:10:40.127875Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.127914Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.127914Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.127924Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.127931Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.127931Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.127998Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037896] server connected, pipe [3:7567326243423178841:2517], now have 1 active actors on pipe
2025-10-31T10:10:40.127998Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037894] server connected, pipe [3:7567326243423178843:2519], now have 1 active actors on pipe
2025-10-31T10:10:40.128059Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.128091Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.128120Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1748: [PQ: 72075186224037896] Created session shared/user_3_1_14607229299462673975_v1 on pipe: [3:7567326243423178841:2517]
2025-10-31T10:10:40.128184Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.128212Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.128226Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037896][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:40.128290Z node 4 :PERSQUEUE DEBUG: partition.cpp:3649: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user session is set to 0 (startOffset 0) session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.128315Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.128436Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037896][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:40.128919Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.128943Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.128954Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.128996Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037892] server connected, pipe [3:7567326243423178842:2518], now have 1 active actors on pipe
2025-10-31T10:10:40.129040Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.129064Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.129084Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1748: [PQ: 72075186224037892] Created session shared/user_3_1_14607229299462673975_v1 on pipe: [3:7567326243423178842:2518]
2025-10-31T10:10:40.129132Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037892] server connected, pipe [3:7567326243423178881:2516], now have 1 active actors on pipe
2025-10-31T10:10:40.129172Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.129184Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.129257Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.129283Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.129293Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037892][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:40.129338Z node 4 :PERSQUEUE DEBUG: partition.cpp:3649: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 user user session is set to 0 (startOffset 0) session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.129350Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.129470Z node 4 :PERSQUEUE INFO: ownerinfo.cpp:30: new Cookie src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 generated for partition 0 topic 'rt3.dc1--test-topic' owner src_id
2025-10-31T10:10:40.129508Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037892][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.129529Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037892][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:40.130991Z node 4 :PQ_READ_PROXY DEBUG: caching_service.cpp:283: Direct read cache: registered server session: shared/user_3_1_14607229299462673975_v1:1 with generation 1
2025-10-31T10:10:40.131046Z node 4 :PQ_READ_PROXY DEBUG: caching_service.cpp:283: Direct read cache: registered server session: shared/user_3_1_14607229299462673975_v1:2 with generation 1
2025-10-31T10:10:40.131758Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.131810Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.131840Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:1748: [PQ: 72075186224037894] Created session shared/user_3_1_14607229299462673975_v1 on pipe: [3:7567326243423178843:2519]
2025-10-31T10:10:40.131913Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.131932Z node 4 :PQ_READ_PROXY DEBUG: caching_service.cpp:283: Direct read cache: registered server session: shared/user_3_1_14607229299462673975_v1:3 with generation 1
2025-10-31T10:10:40.131942Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.131956Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037894][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:40.132028Z node 4 :PERSQUEUE DEBUG: partition.cpp:3649: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user session is set to 0 (startOffset 0) session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.132044Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.132157Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037894][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:40.136048Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:663: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) initDone 0 event { CmdGetClientOffsetResult { Offset: 0 EndOffset: 0 SizeLag: 0 WriteTimestampEstimateMS: 1761905440127 ClientHasAnyCommits: false } Cookie: 18446744073709551615 }
2025-10-31T10:10:40.136103Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:694: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 INIT DONE TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) EndOffset 0 readOffset 0 committedOffset 0
2025-10-31T10:10:40.136189Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:663: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) initDone 0 event { CmdGetClientOffsetResult { Offset: 0 EndOffset: 0 SizeLag: 0 WriteTimestampEstimateMS: 1761905440128 ClientHasAnyCommits: false } Cookie: 18446744073709551615 }
2025-10-31T10:10:40.136212Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:694: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 INIT DONE TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) EndOffset 0 readOffset 0 committedOffset 0
2025-10-31T10:10:40.136397Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:663: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) initDone 0 event { CmdGetClientOffsetResult { Offset: 0 EndOffset: 0 SizeLag: 0 WriteTimestampEstimateMS: 1761905440127 ClientHasAnyCommits: false } Cookie: 18446744073709551615 }
2025-10-31T10:10:40.136421Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:694: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 INIT DONE TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) EndOffset 0 readOffset 0 committedOffset 0
2025-10-31T10:10:40.136484Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1434: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 sending to client partition status
2025-10-31T10:10:40.136805Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1434: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 sending to client partition status
2025-10-31T10:10:40.136946Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1434: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 sending to client partition status
2025-10-31T10:10:40.137578Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Confirm partition stream create. Partition stream id: 1. Cluster: "-". Topic: "test-topic-mirrored-from-dc3". Partition: 0. Read offset: (NULL)
2025-10-31T10:10:40.137814Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Confirm partition stream create. Partition stream id: 2. Cluster: "-". Topic: "test-topic". Partition: 0. Read offset: (NULL)
2025-10-31T10:10:40.138160Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Confirm partition stream create. Partition stream id: 3. Cluster: "-". Topic: "test-topic-mirrored-from-dc2". Partition: 0. Read offset: (NULL)
2025-10-31T10:10:40.135303Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037892][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:40.135317Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037896][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:40.135376Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037896][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:40.135376Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037892][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:40.135396Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.135398Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.135414Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.135416Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.135431Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037892][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:40.135433Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.135462Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:183: [72075186224037896][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:40.135473Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.135495Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 18446744073709551615
2025-10-31T10:10:40.135519Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:35: [72075186224037892][Partition][0][StateIdle] TPartition::ReplyOwnerOk. Partition: 0
2025-10-31T10:10:40.135547Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:183: [72075186224037892][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:40.135576Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 18446744073709551615
2025-10-31T10:10:40.135624Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:40.135640Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037894][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:40.135681Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037894][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:40.135693Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.135702Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.135708Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.135726Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:183: [72075186224037894][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:40.135749Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 18446744073709551615
2025-10-31T10:10:40.137335Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.137360Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.137434Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:40.137999Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:871: session inited cookie: 2 partition: 0 MaxSeqNo: 0 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0
2025-10-31T10:10:40.141901Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1761905440141 BytesWritten: 0 MessagesWritten: 0 BytesWrittenCompressed: 0 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-10-31T10:10:40.142042Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session established. Init response: session_id: "src_id|fd1cdb7c-1262191-d8abc496-99fa060_0" topic: "test-topic" cluster: "dc1" supported_codecs: CODEC_RAW supported_codecs: CODEC_GZIP supported_codecs: CODEC_LZOP
2025-10-31T10:10:40.142259Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write 1 messages with Id from 1 to 1
2025-10-31T10:10:40.142380Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: try to update token
2025-10-31T10:10:40.142422Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Send 1 message(s) (0 left), first sequence number is 1
2025-10-31T10:10:40.142652Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write 1 messages with Id from 2 to 2
2025-10-31T10:10:40.142698Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: try to update token
2025-10-31T10:10:40.142717Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Send 1 message(s) (0 left), first sequence number is 2
2025-10-31T10:10:40.142754Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write 1 messages with Id from 3 to 3
2025-10-31T10:10:40.143298Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: try to update token
2025-10-31T10:10:40.143320Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Send 1 message(s) (0 left), first sequence number is 3
2025-10-31T10:10:40.143437Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write 1 messages with Id from 4 to 4
2025-10-31T10:10:40.143469Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: try to update token
2025-10-31T10:10:40.143490Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Send 1 message(s) (0 left), first sequence number is 4
2025-10-31T10:10:40.143545Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write 1 messages with Id from 5 to 5
2025-10-31T10:10:40.143585Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: try to update token
2025-10-31T10:10:40.143603Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Send 1 message(s) (0 left), first sequence number is 5
2025-10-31T10:10:40.143634Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: close. Timeout = 18446744073709551 ms
2025-10-31T10:10:40.141500Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { start_partition_session_response { partition_session_id: 1 } }
2025-10-31T10:10:40.141671Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:539: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got StartRead from client: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), readOffset# 0, commitOffset# (empty maybe)
2025-10-31T10:10:40.141745Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:1023: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Start reading TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) EndOffset 0 readOffset 0 committedOffset 0 clientCommitOffset (empty maybe) clientReadOffset 0
2025-10-31T10:10:40.141788Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1277: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait data in partition inited, cookie 1 from offset 0
2025-10-31T10:10:40.142084Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { start_partition_session_response { partition_session_id: 2 } }
2025-10-31T10:10:40.142185Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:539: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got StartRead from client: partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), readOffset# 0, commitOffset# (empty maybe)
2025-10-31T10:10:40.142267Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:1023: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Start reading TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) EndOffset 0 readOffset 0 committedOffset 0 clientCommitOffset (empty maybe) clientReadOffset 0
2025-10-31T10:10:40.142295Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1277: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait data in partition inited, cookie 1 from offset 0
2025-10-31T10:10:40.142513Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { start_partition_session_response { partition_session_id: 3 } }
2025-10-31T10:10:40.142592Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:539: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got StartRead from client: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), readOffset# 0, commitOffset# (empty maybe)
2025-10-31T10:10:40.142631Z node 3 :PQ_READ_PROXY INFO: partition_actor.cpp:1023: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Start reading TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) EndOffset 0 readOffset 0 committedOffset 0 clientCommitOffset (empty maybe) clientReadOffset 0
2025-10-31T10:10:40.142656Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1277: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait data in partition inited, cookie 1 from offset 0
2025-10-31T10:10:40.143086Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 2 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.143379Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.144015Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.144133Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 2 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.144290Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 2 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.144398Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.144498Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.144531Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 2 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.144686Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.144978Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 2 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.145174Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.143716Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.143760Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.143826Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-10-31T10:10:40.144841Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.144862Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.145226Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 1 partNo : 0 messageNo: 1 size 91 offset: -1
2025-10-31T10:10:40.145280Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.145291Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.145320Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.145331Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.145364Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.145377Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.145401Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037892][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.145458Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.145482Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.145515Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037892][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:40.145621Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 1 partNo 0
2025-10-31T10:10:40.207379Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.207465Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.207509Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.207546Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.203888Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 1 partNo 0 FormedBlobsCount 0 NewHead: Offset 0 PartNo 0 PackedSize 165 count 1 nextOffset 1 batches 1
2025-10-31T10:10:40.203949Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.204419Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1675: [72075186224037892][Partition][0][StateIdle] Add new write blob: topic 'rt3.dc1--test-topic' partition 0 compactOffset 0,1 HeadOffset 0 endOffset 0 curOffset 1 d0000000000_00000000000000000000_00000_0000000001_00000? size 153 WTime 1761905440204
2025-10-31T10:10:40.204632Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.204649Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.204675Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037892][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:40.204777Z node 4 :PERSQUEUE DEBUG: read.h:313: [72075186224037892][PQCacheProxy]CacheProxy. Passthrough blob. Partition 0 offset 0 partNo 0 count 1 size 153
2025-10-31T10:10:40.204802Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 2  requestId: <none> cookie: 2
2025-10-31T10:10:40.204846Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 3  requestId: <none> cookie: 3
2025-10-31T10:10:40.204858Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 4  requestId: <none> cookie: 4
2025-10-31T10:10:40.204950Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 5  requestId: <none> cookie: 5
2025-10-31T10:10:40.208176Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.208204Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.208258Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 2 partNo : 0 messageNo: 6 size 202 offset: -1
2025-10-31T10:10:40.208321Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.208332Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.208353Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 3 partNo : 0 messageNo: 7 size 312 offset: -1
2025-10-31T10:10:40.208389Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.208400Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.208420Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 4 partNo : 0 messageNo: 8 size 422 offset: -1
2025-10-31T10:10:40.208455Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.208466Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.208484Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 5 partNo : 0 messageNo: 9 size 532 offset: -1
2025-10-31T10:10:40.208509Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037892][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.208551Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037892][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.208566Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037892][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.208578Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037892][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.208594Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037892][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.208610Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037892][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.208620Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037892][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.208631Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037892][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.209236Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:321: Caching head blob in L1. Partition 0 offset 0 count 1 size 153 actorID [4:7567326231825311499:2385]
2025-10-31T10:10:40.209272Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037892][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:40.209312Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037892][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 98 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:40.209341Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037892][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.209368Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037892][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 1, partNo: 0, Offset: 0 is stored on disk
2025-10-31T10:10:40.209562Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:926: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-10-31T10:10:40.209581Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:968: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 user user send read request for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 1 rrg 0
2025-10-31T10:10:40.209607Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.209640Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.209653Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037892][Partition][0][StateIdle] Batch completed (4)
2025-10-31T10:10:40.209739Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 2 partNo 0
2025-10-31T10:10:40.210468Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 2 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 276 count 1 nextOffset 2 batches 1
2025-10-31T10:10:40.210521Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 3 partNo 0
2025-10-31T10:10:40.210548Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 3 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 630 count 2 nextOffset 3 batches 1
2025-10-31T10:10:40.210568Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 4 partNo 0
2025-10-31T10:10:40.210587Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 4 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1094 count 3 nextOffset 4 batches 1
2025-10-31T10:10:40.210609Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 5 partNo 0
2025-10-31T10:10:40.210630Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037892][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 5 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1668 count 4 nextOffset 5 batches 1
2025-10-31T10:10:40.210645Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.211111Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1675: [72075186224037892][Partition][0][StateIdle] Add new write blob: topic 'rt3.dc1--test-topic' partition 0 compactOffset 1,4 HeadOffset 1 endOffset 1 curOffset 5 d0000000000_00000000000000000001_00000_0000000004_00000? size 1600 WTime 1761905440210
2025-10-31T10:10:40.211219Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:193: [72075186224037892][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:40.211275Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 1  requestId: <none> cookie: 1
2025-10-31T10:10:40.211325Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037892][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:40.211429Z node 4 :PERSQUEUE DEBUG: read.h:313: [72075186224037892][PQCacheProxy]CacheProxy. Passthrough blob. Partition 0 offset 1 partNo 0 count 4 size 1600
2025-10-31T10:10:40.211594Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:883: [72075186224037892][Partition][0][StateIdle] read cookie 3 Topic 'rt3.dc1--test-topic' partition 0 user user offset 0 count 1 size 1024000 endOffset 1 max time lag 0ms effective offset 0
2025-10-31T10:10:40.211774Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1097: [72075186224037892][Partition][0][StateIdle] read cookie 3 added 1 blobs, size 153 count 1 last offset 0, current partition end offset: 1
2025-10-31T10:10:40.211790Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1123: [72075186224037892][Partition][0][StateIdle] Reading cookie 3. Send blob request.
2025-10-31T10:10:40.211857Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:497: Got data from cache. Partition 0 offset 0 partno 0 count 1 parts_count 0 source 1 size 153 accessed 0 times before, last time 2025-10-31T10:10:40.000000Z
2025-10-31T10:10:40.211882Z node 4 :PERSQUEUE DEBUG: read.h:126: [72075186224037892][PQCacheProxy]Reading cookie 3. All 1 blobs are from cache.
2025-10-31T10:10:40.211915Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:581: FormAnswer for 1 blobs
2025-10-31T10:10:40.212024Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:496: FormAnswer processing batch offset 0 totakecount 1 count 1 size 133 from pos 0 cbcount 1
2025-10-31T10:10:40.212077Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1018: Topic 'rt3.dc1--test-topic' partition 0 user user readTimeStamp done, result 1761905440144 queuesize 0 startOffset 0
2025-10-31T10:10:40.212704Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:129: PQ Cache (L2). Adding blob. Tablet '72075186224037892' partition 0 offset 0 partno 0 count 1 parts 0 suffix '63' size 153
2025-10-31T10:10:40.212746Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:201: PQ Cache (L2). Touched. Tablet '72075186224037892' partition 0 offset 0 partno 0 count 1 parts 0 suffix '63'
2025-10-31T10:10:40.215370Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1312: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait for data done:  readOffset 0 EndOffset 0 newEndOffset 1 commitOffset 0 clientCommitOffset 0 cookie 1 readingFinished 0 firstRead 1
2025-10-31T10:10:40.215419Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:969: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) ready for read with readOffset 0 endOffset 1
2025-10-31T10:10:40.215483Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.215909Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2332: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), readOffset# 0, endOffset# 1, WTime# 0, sizeLag# 153
2025-10-31T10:10:40.215928Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2343: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1TEvPartitionReady. Aval parts: 1
2025-10-31T10:10:40.215978Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2266: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 performing read request: guid# 46b6a907-be7beb18-2abe6bcb-3cd7b56e, from# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), count# 1, size# 183, partitionsAsked# 1, maxTimeLag# 0ms
2025-10-31T10:10:40.216065Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1395: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 READ FROM TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2)maxCount 1 maxSize 183 maxTimeLagMs 0 readTimestampMs 0 readOffset 0 EndOffset 1 ClientCommitOffset 0 committedOffset 0 Guid 46b6a907-be7beb18-2abe6bcb-3cd7b56e
2025-10-31T10:10:40.217445Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.217488Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.217598Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:883: [72075186224037892][Partition][0][StateIdle] read cookie 4 Topic 'rt3.dc1--test-topic' partition 0 user user offset 0 count 1 size 183 endOffset 1 max time lag 0ms effective offset 0
2025-10-31T10:10:40.217818Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1097: [72075186224037892][Partition][0][StateIdle] read cookie 4 added 1 blobs, size 153 count 1 last offset 0, current partition end offset: 1
2025-10-31T10:10:40.217844Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1123: [72075186224037892][Partition][0][StateIdle] Reading cookie 4. Send blob request.
2025-10-31T10:10:40.217894Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:497: Got data from cache. Partition 0 offset 0 partno 0 count 1 parts_count 0 source 1 size 153 accessed 1 times before, last time 2025-10-31T10:10:40.000000Z
2025-10-31T10:10:40.217923Z node 4 :PERSQUEUE DEBUG: read.h:126: [72075186224037892][PQCacheProxy]Reading cookie 4. All 1 blobs are from cache.
2025-10-31T10:10:40.217963Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:581: FormAnswer for 1 blobs
2025-10-31T10:10:40.218087Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:496: FormAnswer processing batch offset 0 totakecount 1 count 1 size 133 from pos 0 cbcount 1
2025-10-31T10:10:40.218853Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:663: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) initDone 1 event { CmdReadResult { MaxOffset: 1 Result { Offset: 0 Data: "... 91 bytes ..." SourceId: "\000src_id" SeqNo: 1 WriteTimestampMS: 1761905440144 CreateTimestampMS: 1761905440142 UncompressedSize: 20 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 1 SizeLag: 20 RealReadOffset: 0 WaitQuotaTimeMs: 0 EndOffset: 1 StartOffset: 0 } Cookie: 0 }
2025-10-31T10:10:40.218965Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1277: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait data in partition inited, cookie 2 from offset 1
2025-10-31T10:10:40.219001Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:901: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 after read state TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) EndOffset 1 ReadOffset 1 ReadGuid 46b6a907-be7beb18-2abe6bcb-3cd7b56e has messages 1
2025-10-31T10:10:40.219113Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1940: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 read done: guid# 46b6a907-be7beb18-2abe6bcb-3cd7b56e, partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), size# 178
2025-10-31T10:10:40.219145Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2102: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 response to read: guid# 46b6a907-be7beb18-2abe6bcb-3cd7b56e
2025-10-31T10:10:40.219406Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2145: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Process answer. Aval parts: 0
2025-10-31T10:10:40.218190Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:40.218391Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:201: PQ Cache (L2). Touched. Tablet '72075186224037892' partition 0 offset 0 partno 0 count 1 parts 0 suffix '63'
2025-10-31T10:10:40.221353Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session got write response: sequence_numbers: 1 offsets: 0 already_written: false write_statistics { persist_duration_ms: 5 queued_in_partition_duration_ms: 58 }
2025-10-31T10:10:40.221403Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: acknoledged message 1
2025-10-31T10:10:40.221441Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Got ReadResponse, serverBytesSize = 178, now ReadSizeBudget = 0, ReadSizeServerDelta = 8388430
2025-10-31T10:10:40.221556Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8388430
2025-10-31T10:10:40.221756Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 2 (0-0)
2025-10-31T10:10:40.221806Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 178 to budget
2025-10-31T10:10:40.221842Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 178, ReadSizeServerDelta = 8388430
2025-10-31T10:10:40.222166Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-10-31T10:10:40.222240Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (0-0)
>>> event from dataHandler: DataReceived { Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 Message { Data: ..20 bytes.. Information: { Offset: 0 ProducerId: "src_id" SeqNo: 1 CreateTime: 2025-10-31T10:10:40.142000Z WriteTime: 2025-10-31T10:10:40.144000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "_ip": "ipv6:[::1]:33546", "logtype": "unknown", "server": "ipv6:[::1]:33546" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } }
>>> get 1 messages in this event
2025-10-31T10:10:40.223068Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] The application data is transferred to the client. Number of messages 1, size 20 bytes
2025-10-31T10:10:40.223119Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 0 to budget
2025-10-31T10:10:40.224875Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { read_request { bytes_size: 178 } }
2025-10-31T10:10:40.225042Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1839: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got read request: guid# 803f0f53-366ab92a-103401a1-8610794a
2025-10-31T10:10:40.228402Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.228430Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.228443Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.228490Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.228499Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.228506Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.228708Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:321: Caching head blob in L1. Partition 0 offset 1 count 4 size 1600 actorID [4:7567326231825311499:2385]
2025-10-31T10:10:40.228773Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037892][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:40.228818Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037892][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 1496 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:40.228846Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037892][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.228879Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037892][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 2, partNo: 0, Offset: 1 is stored on disk
2025-10-31T10:10:40.228893Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037892][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.228913Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037892][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 3, partNo: 0, Offset: 2 is stored on disk
2025-10-31T10:10:40.228923Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037892][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.228940Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037892][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 4, partNo: 0, Offset: 3 is stored on disk
2025-10-31T10:10:40.228953Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037892][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.228985Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037892][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 5, partNo: 0, Offset: 4 is stored on disk
2025-10-31T10:10:40.229197Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.229213Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.229226Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.229252Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:193: [72075186224037892][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:40.229292Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 6  requestId: <none> cookie: 2
2025-10-31T10:10:40.229325Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 7  requestId: <none> cookie: 3
2025-10-31T10:10:40.229347Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 8  requestId: <none> cookie: 4
2025-10-31T10:10:40.229368Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 9  requestId: <none> cookie: 5
2025-10-31T10:10:40.229394Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.229406Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.229416Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.230102Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:129: PQ Cache (L2). Adding blob. Tablet '72075186224037892' partition 0 offset 1 partno 0 count 4 parts 0 suffix '63' size 1600
2025-10-31T10:10:40.230416Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1312: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait for data done:  readOffset 1 EndOffset 1 newEndOffset 5 commitOffset 0 clientCommitOffset 0 cookie 2 readingFinished 0 firstRead 0
2025-10-31T10:10:40.230457Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:969: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) ready for read with readOffset 1 endOffset 5
2025-10-31T10:10:40.230492Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.230503Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2332: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), readOffset# 1, endOffset# 5, WTime# 1761905440144, sizeLag# 1600
2025-10-31T10:10:40.230519Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2343: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1TEvPartitionReady. Aval parts: 1
2025-10-31T10:10:40.230565Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.230575Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2266: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 performing read request: guid# 68910cd8-fb86bbe6-9765d3a-f45867a, from# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), count# 4, size# 1920, partitionsAsked# 1, maxTimeLag# 0ms
2025-10-31T10:10:40.230595Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.230617Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.230632Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1395: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 READ FROM TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2)maxCount 4 maxSize 1920 maxTimeLagMs 0 readTimestampMs 0 readOffset 1 EndOffset 5 ClientCommitOffset 0 committedOffset 0 Guid 68910cd8-fb86bbe6-9765d3a-f45867a
2025-10-31T10:10:40.231328Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-10-31T10:10:40.231357Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-10-31T10:10:40.231448Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:883: [72075186224037892][Partition][0][StateIdle] read cookie 5 Topic 'rt3.dc1--test-topic' partition 0 user user offset 1 count 4 size 1920 endOffset 5 max time lag 0ms effective offset 1
2025-10-31T10:10:40.231648Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1097: [72075186224037892][Partition][0][StateIdle] read cookie 5 added 1 blobs, size 1600 count 4 last offset 1, current partition end offset: 5
2025-10-31T10:10:40.231663Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1123: [72075186224037892][Partition][0][StateIdle] Reading cookie 5. Send blob request.
2025-10-31T10:10:40.231703Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:497: Got data from cache. Partition 0 offset 1 partno 0 count 4 parts_count 0 source 1 size 1600 accessed 0 times before, last time 2025-10-31T10:10:40.000000Z
2025-10-31T10:10:40.231733Z node 4 :PERSQUEUE DEBUG: read.h:126: [72075186224037892][PQCacheProxy]Reading cookie 5. All 1 blobs are from cache.
2025-10-31T10:10:40.231771Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:581: FormAnswer for 1 blobs
2025-10-31T10:10:40.232013Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:496: FormAnswer processing batch offset 1 totakecount 4 count 4 size 1580 from pos 0 cbcount 4
2025-10-31T10:10:40.232128Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-10-31T10:10:40.232369Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:201: PQ Cache (L2). Touched. Tablet '72075186224037892' partition 0 offset 1 partno 0 count 4 parts 0 suffix '63'
2025-10-31T10:10:40.232942Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:663: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) initDone 1 event { CmdReadResult { MaxOffset: 5 Result { Offset: 1 Data: "... 202 bytes ..." SourceId: "\000src_id" SeqNo: 2 WriteTimestampMS: 1761905440209 CreateTimestampMS: 1761905440142 UncompressedSize: 130 PartitionKey: "" ExplicitHash: "" } Result { Offset: 2 Data: "... 312 bytes ..." SourceId: "\000src_id" SeqNo: 3 WriteTimestampMS: 1761905440210 CreateTimestampMS: 1761905440142 UncompressedSize: 240 PartitionKey: "" ExplicitHash: "" } Result { Offset: 3 Data: "... 422 bytes ..." SourceId: "\000src_id" SeqNo: 4 WriteTimestampMS: 1761905440210 CreateTimestampMS: 1761905440143 UncompressedSize: 350 PartitionKey: "" ExplicitHash: "" } Result { Offset: 4 Data: "... 532 bytes ..." SourceId: "\000src_id" SeqNo: 5 WriteTimestampMS: 1761905440210 CreateTimestampMS: 1761905440143 UncompressedSize: 460 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 1 SizeLag: 18446744073709551580 RealReadOffset: 4 WaitQuotaTimeMs: 0 EndOffset: 5 StartOffset: 0 } Cookie: 1 }
2025-10-31T10:10:40.233546Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session got write response: sequence_numbers: 2 offsets: 1 already_written: false write_statistics { persist_duration_ms: 18 queued_in_partition_duration_ms: 2 }
2025-10-31T10:10:40.233628Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: acknoledged message 2
2025-10-31T10:10:40.233799Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Got ReadResponse, serverBytesSize = 1562, now ReadSizeBudget = 0, ReadSizeServerDelta = 8387046
2025-10-31T10:10:40.233866Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session got write response: sequence_numbers: 3 offsets: 2 already_written: false write_statistics { persist_duration_ms: 18 queued_in_partition_duration_ms: 2 }
2025-10-31T10:10:40.233896Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: acknoledged message 3
2025-10-31T10:10:40.233921Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8387046
2025-10-31T10:10:40.234131Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session got write response: sequence_numbers: 4 offsets: 3 already_written: false write_statistics { persist_duration_ms: 18 queued_in_partition_duration_ms: 2 }
2025-10-31T10:10:40.234179Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: acknoledged message 4
2025-10-31T10:10:40.234179Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 2 (1-4)
2025-10-31T10:10:40.234219Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 1562 to budget
2025-10-31T10:10:40.234254Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 1562, ReadSizeServerDelta = 8387046
2025-10-31T10:10:40.234475Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-10-31T10:10:40.233164Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1277: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait data in partition inited, cookie 3 from offset 5
2025-10-31T10:10:40.233206Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:901: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 after read state TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) EndOffset 5 ReadOffset 5 ReadGuid 68910cd8-fb86bbe6-9765d3a-f45867a has messages 1
2025-10-31T10:10:40.234539Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (1-1)
2025-10-31T10:10:40.233339Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1940: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 read done: guid# 68910cd8-fb86bbe6-9765d3a-f45867a, partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), size# 1562
2025-10-31T10:10:40.233367Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2102: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 response to read: guid# 68910cd8-fb86bbe6-9765d3a-f45867a
2025-10-31T10:10:40.234579Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 0} (2-2)
2025-10-31T10:10:40.233547Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2145: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Process answer. Aval parts: 0
2025-10-31T10:10:40.234676Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 1} (3-3)
2025-10-31T10:10:40.234707Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 2} (4-4)
2025-10-31T10:10:40.234996Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session got write response: sequence_numbers: 5 offsets: 4 already_written: false write_statistics { persist_duration_ms: 18 queued_in_partition_duration_ms: 2 }
2025-10-31T10:10:40.235036Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: acknoledged message 5
2025-10-31T10:10:40.235169Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { read_request { bytes_size: 1562 } }
2025-10-31T10:10:40.235301Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1839: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got read request: guid# 1358fb64-52ab232d-3bc97b6b-753830ad
>>> event from dataHandler: DataReceived { Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 Message { Data: ..130 bytes.. Information: { Offset: 1 ProducerId: "src_id" SeqNo: 2 CreateTime: 2025-10-31T10:10:40.142000Z WriteTime: 2025-10-31T10:10:40.209000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:33546", "_ip": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } Message { Data: ..240 bytes.. Information: { Offset: 2 ProducerId: "src_id" SeqNo: 3 CreateTime: 2025-10-31T10:10:40.142000Z WriteTime: 2025-10-31T10:10:40.210000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "_ip": "ipv6:[::1]:33546", "server": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } Message { Data: ..350 bytes.. Information: { Offset: 3 ProducerId: "src_id" SeqNo: 4 CreateTime: 2025-10-31T10:10:40.143000Z WriteTime: 2025-10-31T10:10:40.210000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "_ip": "ipv6:[::1]:33546", "server": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } Message { Data: ..460 bytes.. Information: { Offset: 4 ProducerId: "src_id" SeqNo: 5 CreateTime: 2025-10-31T10:10:40.143000Z WriteTime: 2025-10-31T10:10:40.210000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "_ip": "ipv6:[::1]:33546", "server": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } }
>>> get 4 messages in this event
2025-10-31T10:10:40.237471Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] The application data is transferred to the client. Number of messages 4, size 1180 bytes
2025-10-31T10:10:40.237524Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 0 to budget
2025-10-31T10:10:40.243777Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session will now close
2025-10-31T10:10:40.243840Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: aborting
2025-10-31T10:10:40.244382Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: gracefully shut down, all writes complete
>>> Writes to test-topic successful
2025-10-31T10:10:40.244432Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fd1cdb7c-1262191-d8abc496-99fa060_0] Write session: destroy
2025-10-31T10:10:40.244890Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: try to update token
2025-10-31T10:10:40.244982Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session: Do CDS request
2025-10-31T10:10:40.245027Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Start write session. Will connect to endpoint: localhost:3655
2025-10-31T10:10:40.246031Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 2 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 grpc read done: success: 0 data: 
2025-10-31T10:10:40.246060Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:256: session v1 cookie: 2 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 grpc read failed
2025-10-31T10:10:40.246092Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:232: session v1 cookie: 2 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 grpc closed
2025-10-31T10:10:40.246110Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:304: session v1 cookie: 2 sessionId: src_id|fd1cdb7c-1262191-d8abc496-99fa060_0 is DEAD
2025-10-31T10:10:40.246790Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::TEvents::TEvPoison
2025-10-31T10:10:40.247452Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2746: [PQ: 72075186224037892] server disconnected, pipe [3:7567326243423178881:2516] destroyed
2025-10-31T10:10:40.247504Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:138: [72075186224037892][Partition][0][StateIdle] TPartition::DropOwner.
2025-10-31T10:10:40.247533Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.247545Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.247559Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.251149Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: send init request: init_request { topic: "test-topic-mirrored-from-dc2" message_group_id: "src_id" }
2025-10-31T10:10:40.251567Z node 3 :PQ_WRITE_PROXY DEBUG: grpc_pq_write.h:107: new grpc connection
2025-10-31T10:10:40.251597Z node 3 :PQ_WRITE_PROXY DEBUG: grpc_pq_write.h:141: new session created cookie 3
2025-10-31T10:10:40.252119Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 3 sessionId:  grpc read done: success: 1 data: init_request { topic: "test-topic-mirrored-from-dc2" message_group_id: "src_id" }
2025-10-31T10:10:40.252281Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:447: session request cookie: 3 topic: "test-topic-mirrored-from-dc2" message_group_id: "src_id" from ipv6:[::1]:33546
2025-10-31T10:10:40.252306Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:1538: write session:  cookie=3 sessionId= userAgent="pqv1 server" ip=ipv6:[::1]:33546 proto=v1  topic=test-topic-mirrored-from-dc2 durationSec=0
2025-10-31T10:10:40.252316Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:571: init check schema
2025-10-31T10:10:40.253938Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:632: session v1 cookie: 3 sessionId:  describe result for acl check
2025-10-31T10:10:40.254074Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:62: TTableHelper SelectQuery: --!syntax_v1
DECLARE $Hash AS Uint32; DECLARE $Topic AS Utf8; DECLARE $SourceId AS Utf8;
SELECT Partition, CreateTime, AccessTime, SeqNo FROM `/Root/PQ/SourceIdMeta2` WHERE Hash == $Hash AND Topic == $Topic AND SourceId == $SourceId;
2025-10-31T10:10:40.254085Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:63: TTableHelper UpdateQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;DECLARE $SeqNo AS Uint64;
UPSERT INTO `/Root/PQ/SourceIdMeta2` (Hash, Topic, SourceId, CreateTime, AccessTime, Partition, SeqNo) VALUES ($Hash, $Topic, $SourceId, $CreateTime, $AccessTime, $Partition, $SeqNo);
2025-10-31T10:10:40.254093Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:64: TTableHelper UpdateAccessTimeQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;
UPDATE `/Root/PQ/SourceIdMeta2` SET AccessTime = $AccessTime WHERE Hash = $Hash AND Topic = $Topic AND SourceId = $SourceId AND Partition = $Partition;
2025-10-31T10:10:40.254109Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:111: TPartitionChooser [3:7567326247718146210:2526] (SourceId=src_id, PreferedPartition=(NULL)) StartKqpSession
2025-10-31T10:10:40.256563Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:142: TPartitionChooser [3:7567326247718146210:2526] (SourceId=src_id, PreferedPartition=(NULL)) Select from the table
2025-10-31T10:10:40.267191Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__old_chooser_actor.h:67: TPartitionChooser [3:7567326247718146210:2526] (SourceId=src_id, PreferedPartition=(NULL)) RequestPQRB
2025-10-31T10:10:40.267635Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7567326247718146229:2526] connected; active server actors: 1
2025-10-31T10:10:40.267774Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__old_chooser_actor.h:80: TPartitionChooser [3:7567326247718146210:2526] (SourceId=src_id, PreferedPartition=(NULL)) Received partition 0 from PQRB for SourceId=src_id
2025-10-31T10:10:40.267805Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:174: TPartitionChooser [3:7567326247718146210:2526] (SourceId=src_id, PreferedPartition=(NULL)) Update the table
2025-10-31T10:10:40.268553Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1659: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7567326247718146229:2526] disconnected.
2025-10-31T10:10:40.268573Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1670: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7567326247718146229:2526] disconnected; active server actors: 1
2025-10-31T10:10:40.268586Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1695: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7567326247718146229:2526] disconnected no session
2025-10-31T10:10:40.275311Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:183: TPartitionChooser [3:7567326247718146210:2526] (SourceId=src_id, PreferedPartition=(NULL)) HandleUpdate PartitionPersisted=0 Status=SUCCESS
2025-10-31T10:10:40.275345Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:305: TPartitionChooser [3:7567326247718146210:2526] (SourceId=src_id, PreferedPartition=(NULL)) ReplyResult: Partition=0, SeqNo=(NULL)
2025-10-31T10:10:40.275356Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:268: TPartitionChooser [3:7567326247718146210:2526] (SourceId=src_id, PreferedPartition=(NULL)) Start idle
2025-10-31T10:10:40.275377Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:694: ProceedPartition. session cookie: 3 sessionId:  partition: 0 expectedGeneration: (NULL)
2025-10-31T10:10:40.277555Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:822: TPartitionWriter 72075186224037894 (partition=0) TEvClientConnected Status OK, TabletId: 72075186224037894, NodeId 4, Generation: 1
2025-10-31T10:10:40.281190Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037894] server connected, pipe [3:7567326247718146245:2526], now have 1 active actors on pipe
2025-10-31T10:10:40.281272Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.281300Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.281386Z node 4 :PERSQUEUE INFO: ownerinfo.cpp:30: new Cookie src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 generated for partition 0 topic 'rt3.dc1--test-topic-mirrored-from-dc2' owner src_id
2025-10-31T10:10:40.281425Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.281445Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.281457Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037894][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:40.281499Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.281540Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:35: [72075186224037894][Partition][0][StateIdle] TPartition::ReplyOwnerOk. Partition: 0
2025-10-31T10:10:40.281592Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:40.286211Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.286254Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.286335Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:40.286808Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:871: session inited cookie: 3 partition: 0 MaxSeqNo: 0 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0
2025-10-31T10:10:40.287682Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1761905440287 BytesWritten: 0 MessagesWritten: 0 BytesWrittenCompressed: 0 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-10-31T10:10:40.287811Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session established. Init response: session_id: "src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0" topic: "test-topic-mirrored-from-dc2" cluster: "dc1" supported_codecs: CODEC_RAW supported_codecs: CODEC_GZIP supported_codecs: CODEC_LZOP
2025-10-31T10:10:40.288130Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write 1 messages with Id from 1 to 1
2025-10-31T10:10:40.288240Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: try to update token
2025-10-31T10:10:40.288278Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Send 1 message(s) (0 left), first sequence number is 1
2025-10-31T10:10:40.288510Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write 1 messages with Id from 2 to 2
2025-10-31T10:10:40.288558Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: try to update token
2025-10-31T10:10:40.288580Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Send 1 message(s) (0 left), first sequence number is 2
2025-10-31T10:10:40.288719Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write 1 messages with Id from 3 to 3
2025-10-31T10:10:40.288768Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: try to update token
2025-10-31T10:10:40.288788Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Send 1 message(s) (0 left), first sequence number is 3
2025-10-31T10:10:40.288885Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write 1 messages with Id from 4 to 4
2025-10-31T10:10:40.288930Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: try to update token
2025-10-31T10:10:40.288948Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Send 1 message(s) (0 left), first sequence number is 4
2025-10-31T10:10:40.289088Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write 1 messages with Id from 5 to 5
2025-10-31T10:10:40.289143Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: try to update token
2025-10-31T10:10:40.289163Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Send 1 message(s) (0 left), first sequence number is 5
2025-10-31T10:10:40.289284Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: close. Timeout = 18446744073709551 ms
2025-10-31T10:10:40.289414Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 3 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.289615Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 3 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.289754Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.289797Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 3 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.289970Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.290040Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.290637Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.290677Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.290717Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.290731Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.290801Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-10-31T10:10:40.290837Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 1  requestId: <none> cookie: 2
2025-10-31T10:10:40.291156Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 3 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.291187Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.291246Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.291484Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.291981Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.292007Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.292047Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.292060Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.292377Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 1 partNo : 0 messageNo: 3 size 91 offset: -1
2025-10-31T10:10:40.292426Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.292439Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.292462Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 2 partNo : 0 messageNo: 4 size 202 offset: -1
2025-10-31T10:10:40.292493Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.292503Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.292547Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037894][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.292593Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.292620Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.292632Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037894][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:40.292737Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 1 partNo 0
2025-10-31T10:10:40.293564Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 1 partNo 0 FormedBlobsCount 0 NewHead: Offset 0 PartNo 0 PackedSize 165 count 1 nextOffset 1 batches 1
2025-10-31T10:10:40.293602Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.294135Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 3 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.294387Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.294057Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1675: [72075186224037894][Partition][0][StateIdle] Add new write blob: topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 compactOffset 0,1 HeadOffset 0 endOffset 0 curOffset 1 d0000000000_00000000000000000000_00000_0000000001_00000? size 153 WTime 1761905440293
2025-10-31T10:10:40.294197Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037894][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.294229Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037894][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.294266Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 2  requestId: <none> cookie: 3
2025-10-31T10:10:40.294318Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037894][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:40.294416Z node 4 :PERSQUEUE DEBUG: read.h:313: [72075186224037894][PQCacheProxy]CacheProxy. Passthrough blob. Partition 0 offset 0 partNo 0 count 1 size 153
2025-10-31T10:10:40.294448Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 5  requestId: <none> cookie: 4
2025-10-31T10:10:40.296739Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.296813Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.297299Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.297333Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.297389Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.297401Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.297452Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 3 partNo : 0 messageNo: 7 size 312 offset: -1
2025-10-31T10:10:40.297514Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.297529Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.297551Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 4 partNo : 0 messageNo: 8 size 422 offset: -1
2025-10-31T10:10:40.297609Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037894][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.297644Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037894][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.297659Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037894][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.297673Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037894][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.297691Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 6  requestId: <none> cookie: 5
2025-10-31T10:10:40.298684Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.299111Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.299136Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.299176Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 5 partNo : 0 messageNo: 9 size 532 offset: -1
2025-10-31T10:10:40.299204Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037894][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.299233Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037894][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.303113Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:321: Caching head blob in L1. Partition 0 offset 0 count 1 size 153 actorID [4:7567326236120278905:2397]
2025-10-31T10:10:40.303177Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037894][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:40.303228Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037894][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 98 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:40.303260Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037894][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.303302Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037894][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 1, partNo: 0, Offset: 0 is stored on disk
2025-10-31T10:10:40.303520Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:926: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-10-31T10:10:40.303549Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:968: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user send read request for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 1 rrg 0
2025-10-31T10:10:40.303573Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.303615Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.303626Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037894][Partition][0][StateIdle] Batch completed (4)
2025-10-31T10:10:40.303723Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 2 partNo 0
2025-10-31T10:10:40.304478Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 2 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 276 count 1 nextOffset 2 batches 1
2025-10-31T10:10:40.304524Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 3 partNo 0
2025-10-31T10:10:40.304554Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 3 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 630 count 2 nextOffset 3 batches 1
2025-10-31T10:10:40.304576Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 4 partNo 0
2025-10-31T10:10:40.304597Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 4 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1094 count 3 nextOffset 4 batches 1
2025-10-31T10:10:40.304620Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 5 partNo 0
2025-10-31T10:10:40.304641Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037894][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 5 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1668 count 4 nextOffset 5 batches 1
2025-10-31T10:10:40.304661Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.305013Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:129: PQ Cache (L2). Adding blob. Tablet '72075186224037894' partition 0 offset 0 partno 0 count 1 parts 0 suffix '63' size 153
2025-10-31T10:10:40.305163Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1675: [72075186224037894][Partition][0][StateIdle] Add new write blob: topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 compactOffset 1,4 HeadOffset 1 endOffset 1 curOffset 5 d0000000000_00000000000000000001_00000_0000000004_00000? size 1600 WTime 1761905440304
2025-10-31T10:10:40.305292Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:193: [72075186224037894][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:40.305347Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 3  requestId: <none> cookie: 1
2025-10-31T10:10:40.305414Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037894][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:40.305520Z node 4 :PERSQUEUE DEBUG: read.h:313: [72075186224037894][PQCacheProxy]CacheProxy. Passthrough blob. Partition 0 offset 1 partNo 0 count 4 size 1600
2025-10-31T10:10:40.305710Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1312: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait for data done:  readOffset 0 EndOffset 0 newEndOffset 1 commitOffset 0 clientCommitOffset 0 cookie 1 readingFinished 0 firstRead 1
2025-10-31T10:10:40.305748Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:969: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) ready for read with readOffset 0 endOffset 1
2025-10-31T10:10:40.305836Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2332: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), readOffset# 0, endOffset# 1, WTime# 0, sizeLag# 153
2025-10-31T10:10:40.305854Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2343: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1TEvPartitionReady. Aval parts: 1
2025-10-31T10:10:40.305901Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2266: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 performing read request: guid# 43fd37b7-1bf94afa-7135f154-78d5e473, from# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), count# 1, size# 183, partitionsAsked# 1, maxTimeLag# 0ms
2025-10-31T10:10:40.305979Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1395: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 READ FROM TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3)maxCount 1 maxSize 183 maxTimeLagMs 0 readTimestampMs 0 readOffset 0 EndOffset 1 ClientCommitOffset 0 committedOffset 0 Guid 43fd37b7-1bf94afa-7135f154-78d5e473
2025-10-31T10:10:40.306128Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.306918Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session got write response: sequence_numbers: 1 offsets: 0 already_written: false write_statistics { persist_duration_ms: 4 queued_in_partition_duration_ms: 1 }
2025-10-31T10:10:40.306985Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: acknoledged message 1
2025-10-31T10:10:40.305757Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:883: [72075186224037894][Partition][0][StateIdle] read cookie 3 Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user offset 0 count 1 size 1024000 endOffset 1 max time lag 0ms effective offset 0
2025-10-31T10:10:40.305929Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1097: [72075186224037894][Partition][0][StateIdle] read cookie 3 added 1 blobs, size 153 count 1 last offset 0, current partition end offset: 1
2025-10-31T10:10:40.305947Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1123: [72075186224037894][Partition][0][StateIdle] Reading cookie 3. Send blob request.
2025-10-31T10:10:40.306022Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:497: Got data from cache. Partition 0 offset 0 partno 0 count 1 parts_count 0 source 1 size 153 accessed 0 times before, last time 2025-10-31T10:10:40.000000Z
2025-10-31T10:10:40.306044Z node 4 :PERSQUEUE DEBUG: read.h:126: [72075186224037894][PQCacheProxy]Reading cookie 3. All 1 blobs are from cache.
2025-10-31T10:10:40.306080Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:581: FormAnswer for 1 blobs
2025-10-31T10:10:40.306193Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:496: FormAnswer processing batch offset 0 totakecount 1 count 1 size 133 from pos 0 cbcount 1
2025-10-31T10:10:40.306256Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1018: Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user readTimeStamp done, result 1761905440292 queuesize 0 startOffset 0
2025-10-31T10:10:40.306461Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:201: PQ Cache (L2). Touched. Tablet '72075186224037894' partition 0 offset 0 partno 0 count 1 parts 0 suffix '63'
2025-10-31T10:10:40.306775Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.306811Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.306898Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:883: [72075186224037894][Partition][0][StateIdle] read cookie 4 Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user offset 0 count 1 size 183 endOffset 1 max time lag 0ms effective offset 0
2025-10-31T10:10:40.307061Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1097: [72075186224037894][Partition][0][StateIdle] read cookie 4 added 1 blobs, size 153 count 1 last offset 0, current partition end offset: 1
2025-10-31T10:10:40.307075Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1123: [72075186224037894][Partition][0][StateIdle] Reading cookie 4. Send blob request.
2025-10-31T10:10:40.307106Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:497: Got data from cache. Partition 0 offset 0 partno 0 count 1 parts_count 0 source 1 size 153 accessed 1 times before, last time 2025-10-31T10:10:40.000000Z
2025-10-31T10:10:40.307131Z node 4 :PERSQUEUE DEBUG: read.h:126: [72075186224037894][PQCacheProxy]Reading cookie 4. All 1 blobs are from cache.
2025-10-31T10:10:40.307161Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:581: FormAnswer for 1 blobs
2025-10-31T10:10:40.307254Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:496: FormAnswer processing batch offset 0 totakecount 1 count 1 size 133 from pos 0 cbcount 1
2025-10-31T10:10:40.307333Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:40.307578Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:201: PQ Cache (L2). Touched. Tablet '72075186224037894' partition 0 offset 0 partno 0 count 1 parts 0 suffix '63'
2025-10-31T10:10:40.309432Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:663: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) initDone 1 event { CmdReadResult { MaxOffset: 1 Result { Offset: 0 Data: "... 91 bytes ..." SourceId: "\000src_id" SeqNo: 1 WriteTimestampMS: 1761905440292 CreateTimestampMS: 1761905440288 UncompressedSize: 20 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 1 SizeLag: 20 RealReadOffset: 0 WaitQuotaTimeMs: 0 EndOffset: 1 StartOffset: 0 } Cookie: 0 }
2025-10-31T10:10:40.309546Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1277: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait data in partition inited, cookie 2 from offset 1
2025-10-31T10:10:40.309591Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:901: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 after read state TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) EndOffset 1 ReadOffset 1 ReadGuid 43fd37b7-1bf94afa-7135f154-78d5e473 has messages 1
2025-10-31T10:10:40.310276Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Got ReadResponse, serverBytesSize = 180, now ReadSizeBudget = 0, ReadSizeServerDelta = 8388428
2025-10-31T10:10:40.310375Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8388428
2025-10-31T10:10:40.310544Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 3 (0-0)
2025-10-31T10:10:40.310588Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 180 to budget
2025-10-31T10:10:40.310621Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 180, ReadSizeServerDelta = 8388428
2025-10-31T10:10:40.310774Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-10-31T10:10:40.310830Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (0-0)
>>> event from dataHandler: DataReceived { Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 Message { Data: ..20 bytes.. Information: { Offset: 0 ProducerId: "src_id" SeqNo: 1 CreateTime: 2025-10-31T10:10:40.288000Z WriteTime: 2025-10-31T10:10:40.292000Z MessageGroupId: "src_id" Meta: { "server": "ipv6:[::1]:33546", "logtype": "unknown", "_ip": "ipv6:[::1]:33546", "ident": "unknown" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } }
>>> get 1 messages in this event
2025-10-31T10:10:40.311056Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] The application data is transferred to the client. Number of messages 1, size 20 bytes
2025-10-31T10:10:40.309672Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1940: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 read done: guid# 43fd37b7-1bf94afa-7135f154-78d5e473, partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), size# 180
2025-10-31T10:10:40.311083Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 0 to budget
2025-10-31T10:10:40.309703Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2102: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 response to read: guid# 43fd37b7-1bf94afa-7135f154-78d5e473
2025-10-31T10:10:40.309917Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2145: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Process answer. Aval parts: 0
2025-10-31T10:10:40.311287Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { read_request { bytes_size: 180 } }
2025-10-31T10:10:40.311382Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1839: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got read request: guid# deece98d-613e8e88-961175d0-46915bc
2025-10-31T10:10:40.313454Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:321: Caching head blob in L1. Partition 0 offset 1 count 4 size 1600 actorID [4:7567326236120278905:2397]
2025-10-31T10:10:40.313517Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037894][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:40.313563Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037894][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 1496 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:40.313592Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037894][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.313604Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:129: PQ Cache (L2). Adding blob. Tablet '72075186224037894' partition 0 offset 1 partno 0 count 4 parts 0 suffix '63' size 1600
2025-10-31T10:10:40.313628Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037894][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 2, partNo: 0, Offset: 1 is stored on disk
2025-10-31T10:10:40.313642Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037894][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.313663Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037894][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 3, partNo: 0, Offset: 2 is stored on disk
2025-10-31T10:10:40.313674Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037894][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.313694Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037894][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 4, partNo: 0, Offset: 3 is stored on disk
2025-10-31T10:10:40.313704Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037894][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.313723Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037894][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 5, partNo: 0, Offset: 4 is stored on disk
2025-10-31T10:10:40.313905Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.313919Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.313930Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.313962Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:193: [72075186224037894][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:40.314000Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 4  requestId: <none> cookie: 2
2025-10-31T10:10:40.314036Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 7  requestId: <none> cookie: 3
2025-10-31T10:10:40.314056Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 8  requestId: <none> cookie: 4
2025-10-31T10:10:40.314079Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 9  requestId: <none> cookie: 5
2025-10-31T10:10:40.316161Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1312: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait for data done:  readOffset 1 EndOffset 1 newEndOffset 5 commitOffset 0 clientCommitOffset 0 cookie 2 readingFinished 0 firstRead 0
2025-10-31T10:10:40.316195Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:969: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) ready for read with readOffset 1 endOffset 5
2025-10-31T10:10:40.316247Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.316304Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.316334Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.316387Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.316812Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2332: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), readOffset# 1, endOffset# 5, WTime# 1761905440292, sizeLag# 1600
2025-10-31T10:10:40.316829Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2343: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1TEvPartitionReady. Aval parts: 1
2025-10-31T10:10:40.316876Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2266: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 performing read request: guid# 3c0e0397-1d639b67-2b65ff93-44c5168a, from# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), count# 4, size# 1920, partitionsAsked# 1, maxTimeLag# 0ms
2025-10-31T10:10:40.316947Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1395: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 READ FROM TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3)maxCount 4 maxSize 1920 maxTimeLagMs 0 readTimestampMs 0 readOffset 1 EndOffset 5 ClientCommitOffset 0 committedOffset 0 Guid 3c0e0397-1d639b67-2b65ff93-44c5168a
2025-10-31T10:10:40.317747Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session got write response: sequence_numbers: 2 offsets: 1 already_written: false write_statistics { persist_duration_ms: 9 queued_in_partition_duration_ms: 9 }
2025-10-31T10:10:40.317799Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: acknoledged message 2
2025-10-31T10:10:40.318274Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session got write response: sequence_numbers: 3 offsets: 2 already_written: false write_statistics { persist_duration_ms: 9 queued_in_partition_duration_ms: 6 }
2025-10-31T10:10:40.318314Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: acknoledged message 3
2025-10-31T10:10:40.318518Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session got write response: sequence_numbers: 4 offsets: 3 already_written: false write_statistics { persist_duration_ms: 9 queued_in_partition_duration_ms: 6 }
2025-10-31T10:10:40.318550Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: acknoledged message 4
2025-10-31T10:10:40.318694Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session got write response: sequence_numbers: 5 offsets: 4 already_written: false write_statistics { persist_duration_ms: 9 queued_in_partition_duration_ms: 5 }
2025-10-31T10:10:40.318724Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: acknoledged message 5
2025-10-31T10:10:40.318375Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-10-31T10:10:40.318415Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-10-31T10:10:40.318525Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:883: [72075186224037894][Partition][0][StateIdle] read cookie 5 Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user offset 1 count 4 size 1920 endOffset 5 max time lag 0ms effective offset 1
2025-10-31T10:10:40.318722Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1097: [72075186224037894][Partition][0][StateIdle] read cookie 5 added 1 blobs, size 1600 count 4 last offset 1, current partition end offset: 5
2025-10-31T10:10:40.318740Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1123: [72075186224037894][Partition][0][StateIdle] Reading cookie 5. Send blob request.
2025-10-31T10:10:40.318785Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:497: Got data from cache. Partition 0 offset 1 partno 0 count 4 parts_count 0 source 1 size 1600 accessed 0 times before, last time 2025-10-31T10:10:40.000000Z
2025-10-31T10:10:40.318823Z node 4 :PERSQUEUE DEBUG: read.h:126: [72075186224037894][PQCacheProxy]Reading cookie 5. All 1 blobs are from cache.
2025-10-31T10:10:40.318867Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:581: FormAnswer for 1 blobs
2025-10-31T10:10:40.319065Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:496: FormAnswer processing batch offset 1 totakecount 4 count 4 size 1580 from pos 0 cbcount 4
2025-10-31T10:10:40.319219Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-10-31T10:10:40.319768Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:201: PQ Cache (L2). Touched. Tablet '72075186224037894' partition 0 offset 1 partno 0 count 4 parts 0 suffix '63'
2025-10-31T10:10:40.320319Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:663: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) initDone 1 event { CmdReadResult { MaxOffset: 5 Result { Offset: 1 Data: "... 202 bytes ..." SourceId: "\000src_id" SeqNo: 2 WriteTimestampMS: 1761905440298 CreateTimestampMS: 1761905440288 UncompressedSize: 130 PartitionKey: "" ExplicitHash: "" } Result { Offset: 2 Data: "... 312 bytes ..." SourceId: "\000src_id" SeqNo: 3 WriteTimestampMS: 1761905440304 CreateTimestampMS: 1761905440288 UncompressedSize: 240 PartitionKey: "" ExplicitHash: "" } Result { Offset: 3 Data: "... 422 bytes ..." SourceId: "\000src_id" SeqNo: 4 WriteTimestampMS: 1761905440304 CreateTimestampMS: 1761905440288 UncompressedSize: 350 PartitionKey: "" ExplicitHash: "" } Result { Offset: 4 Data: "... 532 bytes ..." SourceId: "\000src_id" SeqNo: 5 WriteTimestampMS: 1761905440304 CreateTimestampMS: 1761905440289 UncompressedSize: 460 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 1 SizeLag: 18446744073709551580 RealReadOffset: 4 WaitQuotaTimeMs: 0 EndOffset: 5 StartOffset: 0 } Cookie: 1 }
2025-10-31T10:10:40.320489Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1277: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait data in partition inited, cookie 3 from offset 5
2025-10-31T10:10:40.320523Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:901: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 after read state TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) EndOffset 5 ReadOffset 5 ReadGuid 3c0e0397-1d639b67-2b65ff93-44c5168a has messages 1
2025-10-31T10:10:40.320640Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1940: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 read done: guid# 3c0e0397-1d639b67-2b65ff93-44c5168a, partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), size# 1568
2025-10-31T10:10:40.320666Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2102: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 response to read: guid# 3c0e0397-1d639b67-2b65ff93-44c5168a
2025-10-31T10:10:40.320866Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2145: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Process answer. Aval parts: 0
2025-10-31T10:10:40.321248Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Got ReadResponse, serverBytesSize = 1568, now ReadSizeBudget = 0, ReadSizeServerDelta = 8387040
2025-10-31T10:10:40.321359Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8387040
2025-10-31T10:10:40.322917Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 3 (1-4)
2025-10-31T10:10:40.322958Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 1568 to budget
2025-10-31T10:10:40.322983Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 1568, ReadSizeServerDelta = 8387040
2025-10-31T10:10:40.323141Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-10-31T10:10:40.323191Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (1-1)
2025-10-31T10:10:40.323234Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 0} (2-2)
2025-10-31T10:10:40.323258Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 1} (3-3)
2025-10-31T10:10:40.323287Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 2} (4-4)
>>> event from dataHandler: DataReceived { Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 Message { Data: ..130 bytes.. Information: { Offset: 1 ProducerId: "src_id" SeqNo: 2 CreateTime: 2025-10-31T10:10:40.288000Z WriteTime: 2025-10-31T10:10:40.298000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:33546", "_ip": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } Message { Data: ..240 bytes.. Information: { Offset: 2 ProducerId: "src_id" SeqNo: 3 CreateTime: 2025-10-31T10:10:40.288000Z WriteTime: 2025-10-31T10:10:40.304000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:33546", "_ip": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } Message { Data: ..350 bytes.. Information: { Offset: 3 ProducerId: "src_id" SeqNo: 4 CreateTime: 2025-10-31T10:10:40.288000Z WriteTime: 2025-10-31T10:10:40.304000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:33546", "_ip": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } Message { Data: ..460 bytes.. Information: { Offset: 4 ProducerId: "src_id" SeqNo: 5 CreateTime: 2025-10-31T10:10:40.289000Z WriteTime: 2025-10-31T10:10:40.304000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:33546", "_ip": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } }
>>> get 4 messages in this event
2025-10-31T10:10:40.323816Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] The application data is transferred to the client. Number of messages 4, size 1180 bytes
2025-10-31T10:10:40.323850Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 0 to budget
2025-10-31T10:10:40.323373Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { read_request { bytes_size: 1568 } }
2025-10-31T10:10:40.323470Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1839: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got read request: guid# 19b0dc23-8f7dea77-9f5a2cb-8e35755c
2025-10-31T10:10:40.329400Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.329432Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.329447Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.329508Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.329518Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.329535Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.329564Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.329573Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.329581Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.390368Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session will now close
2025-10-31T10:10:40.390437Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: aborting
2025-10-31T10:10:40.390980Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: gracefully shut down, all writes complete
>>> Writes to test-topic-mirrored-from-dc2 successful
2025-10-31T10:10:40.391032Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0] Write session: destroy
2025-10-31T10:10:40.391463Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: try to update token
2025-10-31T10:10:40.391531Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session: Do CDS request
2025-10-31T10:10:40.391569Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Start write session. Will connect to endpoint: localhost:3655
2025-10-31T10:10:40.392535Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 3 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 grpc read done: success: 0 data: 
2025-10-31T10:10:40.392560Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:256: session v1 cookie: 3 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 grpc read failed
2025-10-31T10:10:40.392592Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:232: session v1 cookie: 3 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 grpc closed
2025-10-31T10:10:40.392612Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:304: session v1 cookie: 3 sessionId: src_id|6d1a08c7-6f4adc79-7c1098f5-dfbf2633_0 is DEAD
2025-10-31T10:10:40.393330Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::TEvents::TEvPoison
2025-10-31T10:10:40.395175Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2746: [PQ: 72075186224037894] server disconnected, pipe [3:7567326247718146245:2526] destroyed
2025-10-31T10:10:40.395238Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:138: [72075186224037894][Partition][0][StateIdle] TPartition::DropOwner.
2025-10-31T10:10:40.395270Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.395280Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.395288Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.397467Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: send init request: init_request { topic: "test-topic-mirrored-from-dc3" message_group_id: "src_id" }
2025-10-31T10:10:40.398167Z node 3 :PQ_WRITE_PROXY DEBUG: grpc_pq_write.h:107: new grpc connection
2025-10-31T10:10:40.398194Z node 3 :PQ_WRITE_PROXY DEBUG: grpc_pq_write.h:141: new session created cookie 4
2025-10-31T10:10:40.400168Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 4 sessionId:  grpc read done: success: 1 data: init_request { topic: "test-topic-mirrored-from-dc3" message_group_id: "src_id" }
2025-10-31T10:10:40.400307Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:447: session request cookie: 4 topic: "test-topic-mirrored-from-dc3" message_group_id: "src_id" from ipv6:[::1]:33546
2025-10-31T10:10:40.400328Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:1538: write session:  cookie=4 sessionId= userAgent="pqv1 server" ip=ipv6:[::1]:33546 proto=v1  topic=test-topic-mirrored-from-dc3 durationSec=0
2025-10-31T10:10:40.400338Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:571: init check schema
2025-10-31T10:10:40.401855Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:632: session v1 cookie: 4 sessionId:  describe result for acl check
2025-10-31T10:10:40.401975Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:62: TTableHelper SelectQuery: --!syntax_v1
DECLARE $Hash AS Uint32; DECLARE $Topic AS Utf8; DECLARE $SourceId AS Utf8;
SELECT Partition, CreateTime, AccessTime, SeqNo FROM `/Root/PQ/SourceIdMeta2` WHERE Hash == $Hash AND Topic == $Topic AND SourceId == $SourceId;
2025-10-31T10:10:40.401990Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:63: TTableHelper UpdateQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;DECLARE $SeqNo AS Uint64;
UPSERT INTO `/Root/PQ/SourceIdMeta2` (Hash, Topic, SourceId, CreateTime, AccessTime, Partition, SeqNo) VALUES ($Hash, $Topic, $SourceId, $CreateTime, $AccessTime, $Partition, $SeqNo);
2025-10-31T10:10:40.401999Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__table_helper.h:64: TTableHelper UpdateAccessTimeQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;
UPDATE `/Root/PQ/SourceIdMeta2` SET AccessTime = $AccessTime WHERE Hash = $Hash AND Topic = $Topic AND SourceId = $SourceId AND Partition = $Partition;
2025-10-31T10:10:40.402015Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:111: TPartitionChooser [3:7567326247718146252:2530] (SourceId=src_id, PreferedPartition=(NULL)) StartKqpSession
2025-10-31T10:10:40.431734Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.431765Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.431781Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.431851Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.431860Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.431869Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.431896Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.431904Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.431911Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.440436Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:142: TPartitionChooser [3:7567326247718146252:2530] (SourceId=src_id, PreferedPartition=(NULL)) Select from the table
2025-10-31T10:10:40.450262Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__old_chooser_actor.h:67: TPartitionChooser [3:7567326247718146252:2530] (SourceId=src_id, PreferedPartition=(NULL)) RequestPQRB
2025-10-31T10:10:40.450913Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1655: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7567326247718146272:2530] connected; active server actors: 1
2025-10-31T10:10:40.451643Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__old_chooser_actor.h:80: TPartitionChooser [3:7567326247718146252:2530] (SourceId=src_id, PreferedPartition=(NULL)) Received partition 0 from PQRB for SourceId=src_id
2025-10-31T10:10:40.451660Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:174: TPartitionChooser [3:7567326247718146252:2530] (SourceId=src_id, PreferedPartition=(NULL)) Update the table
2025-10-31T10:10:40.452994Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1659: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7567326247718146272:2530] disconnected.
2025-10-31T10:10:40.453026Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1670: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7567326247718146272:2530] disconnected; active server actors: 1
2025-10-31T10:10:40.453042Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1695: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7567326247718146272:2530] disconnected no session
2025-10-31T10:10:40.463117Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:183: TPartitionChooser [3:7567326247718146252:2530] (SourceId=src_id, PreferedPartition=(NULL)) HandleUpdate PartitionPersisted=0 Status=SUCCESS
2025-10-31T10:10:40.463155Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:305: TPartitionChooser [3:7567326247718146252:2530] (SourceId=src_id, PreferedPartition=(NULL)) ReplyResult: Partition=0, SeqNo=(NULL)
2025-10-31T10:10:40.463171Z node 3 :PQ_PARTITION_CHOOSER DEBUG: partition_chooser_impl__abstract_chooser_actor.h:268: TPartitionChooser [3:7567326247718146252:2530] (SourceId=src_id, PreferedPartition=(NULL)) Start idle
2025-10-31T10:10:40.463201Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:694: ProceedPartition. session cookie: 4 sessionId:  partition: 0 expectedGeneration: (NULL)
2025-10-31T10:10:40.465539Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2721: [PQ: 72075186224037896] server connected, pipe [3:7567326247718146288:2530], now have 1 active actors on pipe
2025-10-31T10:10:40.467375Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:822: TPartitionWriter 72075186224037896 (partition=0) TEvClientConnected Status OK, TabletId: 72075186224037896, NodeId 4, Generation: 1
2025-10-31T10:10:40.469232Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.469271Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.469359Z node 4 :PERSQUEUE INFO: ownerinfo.cpp:30: new Cookie src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 generated for partition 0 topic 'rt3.dc1--test-topic-mirrored-from-dc3' owner src_id
2025-10-31T10:10:40.469395Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.469412Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.469425Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037896][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:40.469462Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.469502Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:35: [72075186224037896][Partition][0][StateIdle] TPartition::ReplyOwnerOk. Partition: 0
2025-10-31T10:10:40.469548Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:40.473424Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.473462Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.473545Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:40.473880Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:871: session inited cookie: 4 partition: 0 MaxSeqNo: 0 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0
2025-10-31T10:10:40.474601Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1761905440474 BytesWritten: 0 MessagesWritten: 0 BytesWrittenCompressed: 0 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-10-31T10:10:40.474712Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session established. Init response: session_id: "src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0" topic: "test-topic-mirrored-from-dc3" cluster: "dc1" supported_codecs: CODEC_RAW supported_codecs: CODEC_GZIP supported_codecs: CODEC_LZOP
2025-10-31T10:10:40.474958Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write 1 messages with Id from 1 to 1
2025-10-31T10:10:40.475058Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: try to update token
2025-10-31T10:10:40.475096Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Send 1 message(s) (0 left), first sequence number is 1
2025-10-31T10:10:40.475294Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write 1 messages with Id from 2 to 2
2025-10-31T10:10:40.475335Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: try to update token
2025-10-31T10:10:40.475355Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Send 1 message(s) (0 left), first sequence number is 2
2025-10-31T10:10:40.475390Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write 1 messages with Id from 3 to 3
2025-10-31T10:10:40.475420Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: try to update token
2025-10-31T10:10:40.475435Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Send 1 message(s) (0 left), first sequence number is 3
2025-10-31T10:10:40.475462Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write 1 messages with Id from 4 to 4
2025-10-31T10:10:40.475493Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: try to update token
2025-10-31T10:10:40.475509Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Send 1 message(s) (0 left), first sequence number is 4
2025-10-31T10:10:40.475535Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write 1 messages with Id from 5 to 5
2025-10-31T10:10:40.475561Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: try to update token
2025-10-31T10:10:40.475576Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Send 1 message(s) (0 left), first sequence number is 5
2025-10-31T10:10:40.475597Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: close. Timeout = 18446744073709551 ms
2025-10-31T10:10:40.477353Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 4 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.477610Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.478620Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 4 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.478763Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.479094Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 4 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.479218Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.479394Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 4 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.479499Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.479736Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 4 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-10-31T10:10:40.479852Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-10-31T10:10:40.481228Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.481269Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.481318Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.481333Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.481359Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.481369Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.481396Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.481416Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.481443Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.481459Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.481555Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-10-31T10:10:40.481599Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 1  requestId: <none> cookie: 2
2025-10-31T10:10:40.481615Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 2  requestId: <none> cookie: 3
2025-10-31T10:10:40.481630Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 3  requestId: <none> cookie: 4
2025-10-31T10:10:40.481645Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 4  requestId: <none> cookie: 5
2025-10-31T10:10:40.482331Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.482412Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.482437Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.482463Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.482484Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.485237Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.485277Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.485592Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 1 partNo : 0 messageNo: 5 size 91 offset: -1
2025-10-31T10:10:40.485653Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.485667Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.485689Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 2 partNo : 0 messageNo: 6 size 202 offset: -1
2025-10-31T10:10:40.485726Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.485739Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.485757Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 3 partNo : 0 messageNo: 7 size 312 offset: -1
2025-10-31T10:10:40.485791Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.485809Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.485826Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 4 partNo : 0 messageNo: 8 size 422 offset: -1
2025-10-31T10:10:40.485861Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.485872Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.485890Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2011: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 5 partNo : 0 messageNo: 9 size 532 offset: -1
2025-10-31T10:10:40.485918Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037896][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.485972Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.485997Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.486008Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037896][Partition][0][StateIdle] Batch completed (1)
2025-10-31T10:10:40.486112Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 1 partNo 0
2025-10-31T10:10:40.487220Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 1 partNo 0 FormedBlobsCount 0 NewHead: Offset 0 PartNo 0 PackedSize 165 count 1 nextOffset 1 batches 1
2025-10-31T10:10:40.487270Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.487725Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1675: [72075186224037896][Partition][0][StateIdle] Add new write blob: topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 compactOffset 0,1 HeadOffset 0 endOffset 0 curOffset 1 d0000000000_00000000000000000000_00000_0000000001_00000? size 153 WTime 1761905440487
2025-10-31T10:10:40.487856Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037896][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.487889Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037896][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.487903Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037896][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.487918Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037896][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.487930Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037896][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.487945Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037896][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.487957Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:637: [72075186224037896][Partition][0][StateIdle] Received TPartition::TEvWrite
2025-10-31T10:10:40.487973Z node 4 :PERSQUEUE DEBUG: partition.cpp:2251: [72075186224037896][Partition][0][StateIdle] Writing. Can't process transactions and user actions
2025-10-31T10:10:40.487995Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037896][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:40.488090Z node 4 :PERSQUEUE DEBUG: read.h:313: [72075186224037896][PQCacheProxy]CacheProxy. Passthrough blob. Partition 0 offset 0 partNo 0 count 1 size 153
2025-10-31T10:10:40.491607Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:321: Caching head blob in L1. Partition 0 offset 0 count 1 size 153 actorID [4:7567326236120278992:2404]
2025-10-31T10:10:40.491662Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037896][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:40.491708Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037896][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 98 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:40.491738Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037896][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.491771Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037896][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 1, partNo: 0, Offset: 0 is stored on disk
2025-10-31T10:10:40.491970Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:926: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-10-31T10:10:40.491991Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:968: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user send read request for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 1 rrg 0
2025-10-31T10:10:40.492011Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.492044Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.492056Z node 4 :PERSQUEUE DEBUG: partition.cpp:2285: [72075186224037896][Partition][0][StateIdle] Batch completed (4)
2025-10-31T10:10:40.492140Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 2 partNo 0
2025-10-31T10:10:40.492945Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 2 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 276 count 1 nextOffset 2 batches 1
2025-10-31T10:10:40.492990Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 3 partNo 0
2025-10-31T10:10:40.493019Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 3 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 630 count 2 nextOffset 3 batches 1
2025-10-31T10:10:40.493038Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 4 partNo 0
2025-10-31T10:10:40.493059Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 4 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1094 count 3 nextOffset 4 batches 1
2025-10-31T10:10:40.493077Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1319: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 5 partNo 0
2025-10-31T10:10:40.493112Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1423: [72075186224037896][Partition][0][StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 5 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1668 count 4 nextOffset 5 batches 1
2025-10-31T10:10:40.493134Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.493579Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:1675: [72075186224037896][Partition][0][StateIdle] Add new write blob: topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 compactOffset 1,4 HeadOffset 1 endOffset 1 curOffset 5 d0000000000_00000000000000000001_00000_0000000004_00000? size 1600 WTime 1761905440492
2025-10-31T10:10:40.493697Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:193: [72075186224037896][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:40.493745Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 5  requestId: <none> cookie: 1
2025-10-31T10:10:40.493800Z node 4 :PERSQUEUE DEBUG: read.h:275: [72075186224037896][PQCacheProxy]CacheProxy. Passthrough write request to KV
2025-10-31T10:10:40.493910Z node 4 :PERSQUEUE DEBUG: read.h:313: [72075186224037896][PQCacheProxy]CacheProxy. Passthrough blob. Partition 0 offset 1 partNo 0 count 4 size 1600
2025-10-31T10:10:40.494101Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:883: [72075186224037896][Partition][0][StateIdle] read cookie 3 Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user offset 0 count 1 size 1024000 endOffset 1 max time lag 0ms effective offset 0
2025-10-31T10:10:40.494273Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1097: [72075186224037896][Partition][0][StateIdle] read cookie 3 added 1 blobs, size 153 count 1 last offset 0, current partition end offset: 1
2025-10-31T10:10:40.494288Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1123: [72075186224037896][Partition][0][StateIdle] Reading cookie 3. Send blob request.
2025-10-31T10:10:40.494361Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:497: Got data from cache. Partition 0 offset 0 partno 0 count 1 parts_count 0 source 1 size 153 accessed 0 times before, last time 2025-10-31T10:10:40.000000Z
2025-10-31T10:10:40.494382Z node 4 :PERSQUEUE DEBUG: read.h:126: [72075186224037896][PQCacheProxy]Reading cookie 3. All 1 blobs are from cache.
2025-10-31T10:10:40.494417Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:581: FormAnswer for 1 blobs
2025-10-31T10:10:40.494526Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:496: FormAnswer processing batch offset 0 totakecount 1 count 1 size 133 from pos 0 cbcount 1
2025-10-31T10:10:40.494585Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1018: Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user readTimeStamp done, result 1761905440486 queuesize 0 startOffset 0
2025-10-31T10:10:40.494704Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:129: PQ Cache (L2). Adding blob. Tablet '72075186224037896' partition 0 offset 0 partno 0 count 1 parts 0 suffix '63' size 153
2025-10-31T10:10:40.494746Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:201: PQ Cache (L2). Touched. Tablet '72075186224037896' partition 0 offset 0 partno 0 count 1 parts 0 suffix '63'
2025-10-31T10:10:40.497371Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1312: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait for data done:  readOffset 0 EndOffset 0 newEndOffset 1 commitOffset 0 clientCommitOffset 0 cookie 1 readingFinished 0 firstRead 1
2025-10-31T10:10:40.497399Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:969: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) ready for read with readOffset 0 endOffset 1
2025-10-31T10:10:40.497446Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.497872Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2332: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), readOffset# 0, endOffset# 1, WTime# 0, sizeLag# 153
2025-10-31T10:10:40.497890Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2343: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1TEvPartitionReady. Aval parts: 1
2025-10-31T10:10:40.497939Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2266: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 performing read request: guid# 37565267-635b6bae-f39d1e17-12ec04f1, from# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), count# 1, size# 183, partitionsAsked# 1, maxTimeLag# 0ms
2025-10-31T10:10:40.498010Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1395: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 READ FROM TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1)maxCount 1 maxSize 183 maxTimeLagMs 0 readTimestampMs 0 readOffset 0 EndOffset 1 ClientCommitOffset 0 committedOffset 0 Guid 37565267-635b6bae-f39d1e17-12ec04f1
2025-10-31T10:10:40.498697Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session got write response: sequence_numbers: 1 offsets: 0 already_written: false write_statistics { persist_duration_ms: 1 queued_in_partition_duration_ms: 1 }
2025-10-31T10:10:40.498743Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: acknoledged message 1
2025-10-31T10:10:40.498786Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.498820Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.499280Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:883: [72075186224037896][Partition][0][StateIdle] read cookie 4 Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user offset 0 count 1 size 183 endOffset 1 max time lag 0ms effective offset 0
2025-10-31T10:10:40.499450Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1097: [72075186224037896][Partition][0][StateIdle] read cookie 4 added 1 blobs, size 153 count 1 last offset 0, current partition end offset: 1
2025-10-31T10:10:40.499465Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1123: [72075186224037896][Partition][0][StateIdle] Reading cookie 4. Send blob request.
2025-10-31T10:10:40.499505Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:497: Got data from cache. Partition 0 offset 0 partno 0 count 1 parts_count 0 source 1 size 153 accessed 1 times before, last time 2025-10-31T10:10:40.000000Z
2025-10-31T10:10:40.499532Z node 4 :PERSQUEUE DEBUG: read.h:126: [72075186224037896][PQCacheProxy]Reading cookie 4. All 1 blobs are from cache.
2025-10-31T10:10:40.499570Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:581: FormAnswer for 1 blobs
2025-10-31T10:10:40.499677Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:496: FormAnswer processing batch offset 0 totakecount 1 count 1 size 133 from pos 0 cbcount 1
2025-10-31T10:10:40.499766Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-10-31T10:10:40.500245Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:201: PQ Cache (L2). Touched. Tablet '72075186224037896' partition 0 offset 0 partno 0 count 1 parts 0 suffix '63'
2025-10-31T10:10:40.501167Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:663: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) initDone 1 event { CmdReadResult { MaxOffset: 1 Result { Offset: 0 Data: "... 91 bytes ..." SourceId: "\000src_id" SeqNo: 1 WriteTimestampMS: 1761905440486 CreateTimestampMS: 1761905440474 UncompressedSize: 20 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 1 SizeLag: 20 RealReadOffset: 0 WaitQuotaTimeMs: 0 EndOffset: 1 StartOffset: 0 } Cookie: 0 }
2025-10-31T10:10:40.501279Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1277: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait data in partition inited, cookie 2 from offset 1
2025-10-31T10:10:40.501314Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:901: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 after read state TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) EndOffset 1 ReadOffset 1 ReadGuid 37565267-635b6bae-f39d1e17-12ec04f1 has messages 1
2025-10-31T10:10:40.501434Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1940: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 read done: guid# 37565267-635b6bae-f39d1e17-12ec04f1, partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), size# 180
2025-10-31T10:10:40.501461Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2102: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 response to read: guid# 37565267-635b6bae-f39d1e17-12ec04f1
2025-10-31T10:10:40.501689Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2145: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Process answer. Aval parts: 0
2025-10-31T10:10:40.502235Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Got ReadResponse, serverBytesSize = 180, now ReadSizeBudget = 0, ReadSizeServerDelta = 8388428
2025-10-31T10:10:40.502335Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8388428
2025-10-31T10:10:40.502535Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 1 (0-0)
2025-10-31T10:10:40.502579Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 180 to budget
2025-10-31T10:10:40.502624Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 180, ReadSizeServerDelta = 8388428
2025-10-31T10:10:40.502535Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:321: Caching head blob in L1. Partition 0 offset 1 count 4 size 1600 actorID [4:7567326236120278992:2404]
2025-10-31T10:10:40.502584Z node 4 :PERSQUEUE DEBUG: partition.cpp:2099: [72075186224037896][Partition][0][StateIdle] Received TEvKeyValue::TEvResponse
2025-10-31T10:10:40.502636Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:574: [72075186224037896][Partition][0][StateIdle] TPartition::HandleWriteResponse writeNewSize# 1496 WriteNewSizeFromSupportivePartitions# 0
2025-10-31T10:10:40.502663Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037896][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.502698Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037896][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 2, partNo: 0, Offset: 1 is stored on disk
2025-10-31T10:10:40.502713Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037896][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.502744Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037896][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 3, partNo: 0, Offset: 2 is stored on disk
2025-10-31T10:10:40.502755Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037896][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.502795Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-10-31T10:10:40.502776Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037896][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 4, partNo: 0, Offset: 3 is stored on disk
2025-10-31T10:10:40.502786Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:58: [72075186224037896][Partition][0][StateIdle] TPartition::ReplyWrite. Partition: 0
2025-10-31T10:10:40.502804Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:361: [72075186224037896][Partition][0][StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 5, partNo: 0, Offset: 4 is stored on disk
2025-10-31T10:10:40.502852Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (0-0)
>>> event from dataHandler: DataReceived { Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 Message { Data: ..20 bytes.. Information: { Offset: 0 ProducerId: "src_id" SeqNo: 1 CreateTime: 2025-10-31T10:10:40.474000Z WriteTime: 2025-10-31T10:10:40.486000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "logtype": "unknown", "_ip": "ipv6:[::1]:33546", "server": "ipv6:[::1]:33546" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } }
>>> get 1 messages in this event
2025-10-31T10:10:40.503450Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] The application data is transferred to the client. Number of messages 1, size 20 bytes
2025-10-31T10:10:40.503493Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 0 to budget
2025-10-31T10:10:40.502982Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.503000Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.503013Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.503045Z node 4 :PERSQUEUE DEBUG: partition_compaction.cpp:193: [72075186224037896][Partition][0][StateIdle] No data for blobs compaction
2025-10-31T10:10:40.503177Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:129: PQ Cache (L2). Adding blob. Tablet '72075186224037896' partition 0 offset 1 partno 0 count 4 parts 0 suffix '63' size 1600
2025-10-31T10:10:40.503780Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 6  requestId: <none> cookie: 2
2025-10-31T10:10:40.503825Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 7  requestId: <none> cookie: 3
2025-10-31T10:10:40.503843Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 8  requestId: <none> cookie: 4
2025-10-31T10:10:40.503862Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 9  requestId: <none> cookie: 5
2025-10-31T10:10:40.503553Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { read_request { bytes_size: 180 } }
2025-10-31T10:10:40.503658Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1839: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got read request: guid# 24e2dfd-cb273ecb-b9457019-6a812b23
2025-10-31T10:10:40.503723Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1312: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait for data done:  readOffset 1 EndOffset 1 newEndOffset 5 commitOffset 0 clientCommitOffset 0 cookie 2 readingFinished 0 firstRead 0
2025-10-31T10:10:40.503743Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:969: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) ready for read with readOffset 1 endOffset 5
2025-10-31T10:10:40.503783Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2332: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), readOffset# 1, endOffset# 5, WTime# 1761905440486, sizeLag# 1600
2025-10-31T10:10:40.503797Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2343: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1TEvPartitionReady. Aval parts: 1
2025-10-31T10:10:40.503831Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2266: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 performing read request: guid# 54850878-3ade5c9d-1c66367a-dd724d6e, from# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), count# 4, size# 1920, partitionsAsked# 1, maxTimeLag# 0ms
2025-10-31T10:10:40.503877Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1395: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 READ FROM TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1)maxCount 4 maxSize 1920 maxTimeLagMs 0 readTimestampMs 0 readOffset 1 EndOffset 5 ClientCommitOffset 0 committedOffset 0 Guid 54850878-3ade5c9d-1c66367a-dd724d6e
2025-10-31T10:10:40.504317Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:146: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-10-31T10:10:40.504345Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2635: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-10-31T10:10:40.504433Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:883: [72075186224037896][Partition][0][StateIdle] read cookie 5 Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user offset 1 count 4 size 1920 endOffset 5 max time lag 0ms effective offset 1
2025-10-31T10:10:40.504609Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1097: [72075186224037896][Partition][0][StateIdle] read cookie 5 added 1 blobs, size 1600 count 4 last offset 1, current partition end offset: 5
2025-10-31T10:10:40.504624Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:1123: [72075186224037896][Partition][0][StateIdle] Reading cookie 5. Send blob request.
2025-10-31T10:10:40.504661Z node 4 :PERSQUEUE DEBUG: cache_eviction.h:497: Got data from cache. Partition 0 offset 1 partno 0 count 4 parts_count 0 source 1 size 1600 accessed 0 times before, last time 2025-10-31T10:10:40.000000Z
2025-10-31T10:10:40.504689Z node 4 :PERSQUEUE DEBUG: read.h:126: [72075186224037896][PQCacheProxy]Reading cookie 5. All 1 blobs are from cache.
2025-10-31T10:10:40.504729Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:581: FormAnswer for 1 blobs
2025-10-31T10:10:40.504936Z node 4 :PERSQUEUE DEBUG: partition_read.cpp:496: FormAnswer processing batch offset 1 totakecount 4 count 4 size 1580 from pos 0 cbcount 4
2025-10-31T10:10:40.505040Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:181: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-10-31T10:10:40.504292Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.504353Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.504386Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.504419Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-10-31T10:10:40.505524Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session got write response: sequence_numbers: 2 offsets: 1 already_written: false write_statistics { persist_duration_ms: 8 queued_in_partition_duration_ms: 4 }
2025-10-31T10:10:40.505576Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: acknoledged message 2
2025-10-31T10:10:40.505468Z node 4 :PERSQUEUE DEBUG: pq_l2_cache.cpp:201: PQ Cache (L2). Touched. Tablet '72075186224037896' partition 0 offset 1 partno 0 count 4 parts 0 suffix '63'
2025-10-31T10:10:40.505775Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session got write response: sequence_numbers: 3 offsets: 2 already_written: false write_statistics { persist_duration_ms: 8 queued_in_partition_duration_ms: 4 }
2025-10-31T10:10:40.505810Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: acknoledged message 3
2025-10-31T10:10:40.505953Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session got write response: sequence_numbers: 4 offsets: 3 already_written: false write_statistics { persist_duration_ms: 8 queued_in_partition_duration_ms: 4 }
2025-10-31T10:10:40.505973Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: acknoledged message 4
2025-10-31T10:10:40.506094Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session got write response: sequence_numbers: 5 offsets: 4 already_written: false write_statistics { persist_duration_ms: 8 queued_in_partition_duration_ms: 4 }
2025-10-31T10:10:40.506119Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: acknoledged message 5
2025-10-31T10:10:40.506841Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Got ReadResponse, serverBytesSize = 1568, now ReadSizeBudget = 0, ReadSizeServerDelta = 8387040
2025-10-31T10:10:40.506941Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8387040
2025-10-31T10:10:40.507123Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 1 (1-4)
2025-10-31T10:10:40.505964Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:663: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) initDone 1 event { CmdReadResult { MaxOffset: 5 Result { Offset: 1 Data: "... 202 bytes ..." SourceId: "\000src_id" SeqNo: 2 WriteTimestampMS: 1761905440488 CreateTimestampMS: 1761905440475 UncompressedSize: 130 PartitionKey: "" ExplicitHash: "" } Result { Offset: 2 Data: "... 312 bytes ..." SourceId: "\000src_id" SeqNo: 3 WriteTimestampMS: 1761905440492 CreateTimestampMS: 1761905440475 UncompressedSize: 240 PartitionKey: "" ExplicitHash: "" } Result { Offset: 3 Data: "... 422 bytes ..." SourceId: "\000src_id" SeqNo: 4 WriteTimestampMS: 1761905440492 CreateTimestampMS: 1761905440475 UncompressedSize: 350 PartitionKey: "" ExplicitHash: "" } Result { Offset: 4 Data: "... 532 bytes ..." SourceId: "\000src_id" SeqNo: 5 WriteTimestampMS: 1761905440492 CreateTimestampMS: 1761905440475 UncompressedSize: 460 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 1 SizeLag: 18446744073709551580 RealReadOffset: 4 WaitQuotaTimeMs: 0 EndOffset: 5 StartOffset: 0 } Cookie: 1 }
2025-10-31T10:10:40.507165Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 1568 to budget
2025-10-31T10:10:40.506124Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:1277: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait data in partition inited, cookie 3 from offset 5
2025-10-31T10:10:40.506158Z node 3 :PQ_READ_PROXY DEBUG: partition_actor.cpp:901: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 after read state TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) EndOffset 5 ReadOffset 5 ReadGuid 54850878-3ade5c9d-1c66367a-dd724d6e has messages 1
2025-10-31T10:10:40.507189Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] In ContinueReadingDataImpl, ReadSizeBudget = 1568, ReadSizeServerDelta = 8387040
2025-10-31T10:10:40.506263Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1940: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 read done: guid# 54850878-3ade5c9d-1c66367a-dd724d6e, partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), size# 1568
2025-10-31T10:10:40.506290Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2102: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 response to read: guid# 54850878-3ade5c9d-1c66367a-dd724d6e
2025-10-31T10:10:40.506451Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:2145: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 Process answer. Aval parts: 0
2025-10-31T10:10:40.507359Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-10-31T10:10:40.507411Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (1-1)
2025-10-31T10:10:40.507447Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 0} (2-2)
2025-10-31T10:10:40.507483Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 1} (3-3)
2025-10-31T10:10:40.507503Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 2} (4-4)
>>> event from dataHandler: DataReceived { Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 Message { Data: ..130 bytes.. Information: { Offset: 1 ProducerId: "src_id" SeqNo: 2 CreateTime: 2025-10-31T10:10:40.475000Z WriteTime: 2025-10-31T10:10:40.488000Z MessageGroupId: "src_id" Meta: { "logtype": "unknown", "server": "ipv6:[::1]:33546", "_ip": "ipv6:[::1]:33546", "ident": "unknown" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } Message { Data: ..240 bytes.. Information: { Offset: 2 ProducerId: "src_id" SeqNo: 3 CreateTime: 2025-10-31T10:10:40.475000Z WriteTime: 2025-10-31T10:10:40.492000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:33546", "_ip": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } Message { Data: ..350 bytes.. Information: { Offset: 3 ProducerId: "src_id" SeqNo: 4 CreateTime: 2025-10-31T10:10:40.475000Z WriteTime: 2025-10-31T10:10:40.492000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:33546", "_ip": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } Message { Data: ..460 bytes.. Information: { Offset: 4 ProducerId: "src_id" SeqNo: 5 CreateTime: 2025-10-31T10:10:40.475000Z WriteTime: 2025-10-31T10:10:40.492000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:33546", "_ip": "ipv6:[::1]:33546", "logtype": "unknown" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } }
>>> get 4 messages in this event
2025-10-31T10:10:40.507817Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] The application data is transferred to the client. Number of messages 4, size 1180 bytes
2025-10-31T10:10:40.507848Z :DEBUG: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] [] Returning serverBytesSize = 0 to budget
2025-10-31T10:10:40.508171Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 1, data# { read_request { bytes_size: 1568 } }
2025-10-31T10:10:40.508262Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:1839: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 got read request: guid# c2f59069-ecb79f2-2131b8da-6c5befd2
2025-10-31T10:10:40.523115Z :INFO: [/Root] Read/commit by partition streams (cluster:topic:partition:stream-id:read-offset:committed-offset): -:test-topic-mirrored-from-dc2:0:3:4:0 -:test-topic:0:2:4:0 -:test-topic-mirrored-from-dc3:0:1:4:0
2025-10-31T10:10:40.523189Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1002 BytesRead: 3600 MessagesRead: 15 BytesReadCompressed: 3600 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-10-31T10:10:40.532867Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.532907Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.532925Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.533004Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.533015Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.533026Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.533058Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.533078Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.533088Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.577176Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session will now close
2025-10-31T10:10:40.577244Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: aborting
2025-10-31T10:10:40.577759Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: gracefully shut down, all writes complete
>>> Writes to test-topic-mirrored-from-dc3 successful
2025-10-31T10:10:40.577806Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0] Write session: destroy
2025-10-31T10:10:40.577969Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Closing read session. Close timeout: 18446744073709.551615s
2025-10-31T10:10:40.578033Z :INFO: [/Root] Read/commit by partition streams (cluster:topic:partition:stream-id:read-offset:committed-offset): -:test-topic-mirrored-from-dc2:0:3:4:0 -:test-topic:0:2:4:0 -:test-topic-mirrored-from-dc3:0:1:4:0
2025-10-31T10:10:40.578073Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1057 BytesRead: 3600 MessagesRead: 15 BytesReadCompressed: 3600 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-10-31T10:10:40.578550Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Closing read session. Close timeout: 0.000000s
2025-10-31T10:10:40.578596Z :INFO: [/Root] Read/commit by partition streams (cluster:topic:partition:stream-id:read-offset:committed-offset): -:test-topic-mirrored-from-dc2:0:3:4:0 -:test-topic:0:2:4:0 -:test-topic-mirrored-from-dc3:0:1:4:0
2025-10-31T10:10:40.578634Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1058 BytesRead: 3600 MessagesRead: 15 BytesReadCompressed: 3600 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-10-31T10:10:40.578669Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Closing read session. Close timeout: 0.000000s
2025-10-31T10:10:40.578707Z :INFO: [/Root] Read/commit by partition streams (cluster:topic:partition:stream-id:read-offset:committed-offset): -:test-topic-mirrored-from-dc2:0:3:4:0 -:test-topic:0:2:4:0 -:test-topic-mirrored-from-dc3:0:1:4:0
2025-10-31T10:10:40.578753Z :INFO: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1058 BytesRead: 3600 MessagesRead: 15 BytesReadCompressed: 3600 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-10-31T10:10:40.578845Z :NOTICE: [/Root] [/Root] [c1fdb2c4-ed909fd2-cf9b0448-3735a509] Aborting read session. Description: SessionClosed { Status: ABORTED Issues: "<main>: Error: Aborted " }
2025-10-31T10:10:40.582629Z node 3 :PQ_WRITE_PROXY DEBUG: write_session_actor.cpp:254: session v1 cookie: 4 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 grpc read done: success: 0 data: 
2025-10-31T10:10:40.582658Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:256: session v1 cookie: 4 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 grpc read failed
2025-10-31T10:10:40.582691Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:232: session v1 cookie: 4 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 grpc closed
2025-10-31T10:10:40.582709Z node 3 :PQ_WRITE_PROXY INFO: write_session_actor.cpp:304: session v1 cookie: 4 sessionId: src_id|5fa05f08-ef0da15b-d0a22a80-8ca7e41c_0 is DEAD
2025-10-31T10:10:40.583689Z node 3 :PQ_WRITE_PROXY DEBUG: writer.cpp:562: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::TEvents::TEvPoison
2025-10-31T10:10:40.583847Z node 3 :PQ_READ_PROXY DEBUG: read_session_actor.cpp:122: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read done: success# 0, data# {  }
2025-10-31T10:10:40.583860Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:125: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc read failed
2025-10-31T10:10:40.583880Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:92: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 grpc closed
2025-10-31T10:10:40.583935Z node 3 :PQ_READ_PROXY INFO: read_session_actor.cpp:383: session cookie 1 consumer shared/user session shared/user_3_1_14607229299462673975_v1 is DEAD
2025-10-31T10:10:40.585012Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2746: [PQ: 72075186224037896] server disconnected, pipe [3:7567326247718146288:2530] destroyed
2025-10-31T10:10:40.585058Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2258: [PQ: 72075186224037896] Destroy direct read session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.585081Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2746: [PQ: 72075186224037896] server disconnected, pipe [3:7567326243423178841:2517] destroyed
2025-10-31T10:10:40.585132Z node 4 :PERSQUEUE DEBUG: partition_write.cpp:138: [72075186224037896][Partition][0][StateIdle] TPartition::DropOwner.
2025-10-31T10:10:40.585173Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.585189Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.585203Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.585253Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2258: [PQ: 72075186224037894] Destroy direct read session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.585271Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2746: [PQ: 72075186224037894] server disconnected, pipe [3:7567326243423178843:2519] destroyed
2025-10-31T10:10:40.585291Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2258: [PQ: 72075186224037892] Destroy direct read session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.585309Z node 4 :PERSQUEUE DEBUG: pq_impl.cpp:2746: [PQ: 72075186224037892] server disconnected, pipe [3:7567326243423178842:2518] destroyed
2025-10-31T10:10:40.585344Z node 4 :PQ_READ_PROXY DEBUG: caching_service.cpp:139: Direct read cache: server session deregistered: shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.585359Z node 4 :PQ_READ_PROXY DEBUG: caching_service.cpp:139: Direct read cache: server session deregistered: shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.585375Z node 4 :PQ_READ_PROXY DEBUG: caching_service.cpp:139: Direct read cache: server session deregistered: shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.585549Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1659: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7567326243423178831:2513] disconnected.
2025-10-31T10:10:40.585580Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1670: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7567326243423178831:2513] disconnected; active server actors: 1
2025-10-31T10:10:40.585601Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: read_balancer__balancing.cpp:1679: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7567326243423178831:2513] client user disconnected session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.585664Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1659: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326243423178832:2513] disconnected.
2025-10-31T10:10:40.585680Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1670: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326243423178832:2513] disconnected; active server actors: 1
2025-10-31T10:10:40.585691Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: read_balancer__balancing.cpp:1679: [72075186224037893][rt3.dc1--test-topic] pipe [3:7567326243423178832:2513] client user disconnected session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.585730Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: read_balancer__balancing.cpp:1659: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7567326243423178833:2513] disconnected.
2025-10-31T10:10:40.585743Z node 3 :PERSQUEUE_READ_BALANCER INFO: read_balancer__balancing.cpp:1670: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7567326243423178833:2513] disconnected; active server actors: 1
2025-10-31T10:10:40.585754Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: read_balancer__balancing.cpp:1679: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7567326243423178833:2513] client user disconnected session shared/user_3_1_14607229299462673975_v1
2025-10-31T10:10:40.633350Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.633371Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.633381Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.633391Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.633397Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.633413Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.633460Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.633470Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.633480Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.734871Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037894][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.734907Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037894][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.734925Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037894][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.735005Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037896][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.735016Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037896][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.735024Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037896][Partition][0][StateIdle] Try persist
2025-10-31T10:10:40.735054Z node 4 :PERSQUEUE DEBUG: partition.cpp:2272: [72075186224037892][Partition][0][StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-10-31T10:10:40.735074Z node 4 :PERSQUEUE DEBUG: partition.cpp:2281: [72075186224037892][Partition][0][StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-10-31T10:10:40.735084Z node 4 :PERSQUEUE DEBUG: partition.cpp:2304: [72075186224037892][Partition][0][StateIdle] Try persist
2025-10-31T10:10:41.315134Z node 3 :FLAT_TX_SCHEMESHARD WARN: schemeshard_impl.cpp:7727: Cannot get console configs
2025-10-31T10:10:41.315164Z node 3 :IMPORT WARN: schemeshard_import.cpp:307: Table profiles were not loaded
