2025-11-05T14:13:52.431796Z :ReadMirrored INFO: Random seed for debugging is 1762352032431775
2025-11-05T14:13:52.940837Z node 3 :METADATA_PROVIDER WARN: fline=table_exists.cpp:54;actor=TTableExistsActor;event=undelivered;self_id=[3:7569244345377734253:2149];send_to=[0:7307199536658146131:7762515];
2025-11-05T14:13:52.942668Z node 4 :METADATA_PROVIDER WARN: fline=table_exists.cpp:54;actor=TTableExistsActor;event=undelivered;self_id=[4:7569244341998144109:2151];send_to=[0:7307199536658146131:7762515];
2025-11-05T14:13:52.945822Z node 3 :METADATA_PROVIDER ERROR: fline=accessor_snapshot_base.cpp:69;action=cannot detect path existence;path=//Root/.metadata/initialization/migrations;error=scheme_cache_undelivered_message;
2025-11-05T14:13:52.947569Z node 4 :METADATA_PROVIDER ERROR: fline=accessor_snapshot_base.cpp:69;action=cannot detect path existence;path=//Root/.metadata/initialization/migrations;error=scheme_cache_undelivered_message;
test_client.cpp: SetPath # /home/runner/.ya/build/build_root/j3ko/003c11/r3tmp/tmpUJ64le/pdisk_1.dat
2025-11-05T14:13:53.004354Z node 3 :PQ_READ_PROXY DEBUG: Direct read cache: : Created
2025-11-05T14:13:53.051248Z node 4 :PQ_READ_PROXY DEBUG: Direct read cache: : Created
2025-11-05T14:13:53.212787Z node 3 :IMPORT WARN: Table profiles were not loaded
2025-11-05T14:13:53.240992Z node 3 :HIVE WARN: HIVE#72057594037968897 Node(3, (0,0,0,0)) VolatileState: Unknown -> Disconnected
2025-11-05T14:13:53.241115Z node 3 :HIVE WARN: HIVE#72057594037968897 Node(3, (0,0,0,0)) VolatileState: Disconnected -> Connecting
2025-11-05T14:13:53.241463Z node 3 :HIVE WARN: HIVE#72057594037968897 Node(4, (0,0,0,0)) VolatileState: Unknown -> Disconnected
2025-11-05T14:13:53.241516Z node 3 :HIVE WARN: HIVE#72057594037968897 Node(4, (0,0,0,0)) VolatileState: Disconnected -> Connecting
2025-11-05T14:13:53.249317Z node 3 :HIVE WARN: HIVE#72057594037968897 Handle TEvInterconnect::TEvNodeConnected, NodeId 4 Cookie 4
2025-11-05T14:13:53.249464Z node 3 :HIVE WARN: HIVE#72057594037968897 Node(3, (0,0,0,0)) VolatileState: Connecting -> Connected
2025-11-05T14:13:53.250936Z node 3 :HIVE WARN: HIVE#72057594037968897 Node(4, (0,0,0,0)) VolatileState: Connecting -> Connected
TServer::EnableGrpc on GrpcPort 20226, node 3
2025-11-05T14:13:53.370653Z node 3 :NET_CLASSIFIER WARN: distributable config is empty, broken or outdated, will use file: /home/runner/.ya/build/build_root/j3ko/003c11/r3tmp/yandexWzxySi.tmp
2025-11-05T14:13:53.370687Z node 3 :NET_CLASSIFIER WARN: will try to initialize from file: /home/runner/.ya/build/build_root/j3ko/003c11/r3tmp/yandexWzxySi.tmp
2025-11-05T14:13:53.370845Z node 3 :NET_CLASSIFIER WARN: successfully initialized from file: /home/runner/.ya/build/build_root/j3ko/003c11/r3tmp/yandexWzxySi.tmp
2025-11-05T14:13:53.371014Z node 3 :NET_CLASSIFIER ERROR: got bad distributable configuration
2025-11-05T14:13:53.440310Z INFO: TTestServer started on Port 31553 GrpcPort 20226
TClient is connected to server localhost:31553
PQClient connected to localhost:20226
WaitRootIsUp 'Root'...
TClient::Ls request: Root
TClient::Ls response: Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "Root"
    PathId: 1
    SchemeshardId: 72057594046644480
    PathType: EPathTypeDir
    CreateFinished: true
    CreateTxId: 1
    CreateStep: 0
    ParentPathId: 1
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 1
      SubDomainVersion: 0
      SecurityStateVersion: 0
    }
    ChildrenExist: false
  }
  Children {
    Name: ".sys"
    PathId: 18446744073709551615
    SchemeshardId: 72057594046644480
    PathType: EPathTypeDir
    CreateFinished: true
    CreateTxId: 0
    CreateStep: 0
    ParentPathId: 18446744073709551615
  }
  DomainDescription {
    SchemeShardId_Depricated: 72057594046644480
    PathId_Depricated: 1
    ProcessingParams {
      Version: 0
      Pl...
(TRUNCATED)
WaitRootIsUp 'Root' success.
2025-11-05T14:13:53.654275Z node 3 :FLAT_TX_SCHEMESHARD WARN: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpAlterSubDomain, opId: 281474976715657:0, at schemeshard:  72057594046644480
waiting...
waiting...
waiting...
waiting...
2025-11-05T14:13:53.946204Z node 3 :TX_CONVEYOR ERROR: fline=service.h:53;problem=unexpected event for task executor;ev_type=NActors::TEvents::TEvWakeup;
2025-11-05T14:13:53.948934Z node 4 :TX_CONVEYOR ERROR: fline=service.h:53;problem=unexpected event for task executor;ev_type=NActors::TEvents::TEvWakeup;
2025-11-05T14:13:56.770374Z node 4 :KQP_COMPILE_ACTOR ERROR: Compilation failed, self: [4:7569244359178013495:2289], status: SCHEME_ERROR, issues: <main>: Error: Type annotation, code: 1030
    <main>:3:16: Error: At function: KiReadTable!
        <main>:3:16: Error: Cannot find table 'db.[/Root/PQ/Config/V2/Cluster]' because it does not exist or you do not have access permissions. Please check correctness of table path and user permissions., code: 2003
2025-11-05T14:13:56.772375Z node 4 :KQP_SESSION WARN: SessionId: ydb://session/3?node_id=4&id=NjUzZjA1NDMtZmQ0MTkyMDMtYzVmZDkyZDYtM2Q1YzkwY2U=, ActorId: [4:7569244359178013493:2288], ActorState: ExecuteState, TraceId: 01k9a5vyw21bdd80g4v3e4fezp, ReplyQueryCompileError, status SCHEME_ERROR remove tx with tx_id: 
2025-11-05T14:13:56.773017Z node 4 :PERSQUEUE_CLUSTER_TRACKER ERROR: failed to list clusters: { Response { QueryIssues { message: "Type annotation" issue_code: 1030 severity: 1 issues { position { row: 3 column: 16 } message: "At function: KiReadTable!" end_position { row: 3 column: 16 } severity: 1 issues { position { row: 3 column: 16 } message: "Cannot find table \'db.[/Root/PQ/Config/V2/Cluster]\' because it does not exist or you do not have access permissions. Please check correctness of table path and user permissions." end_position { row: 3 column: 16 } issue_code: 2003 severity: 1 } } } TxMeta { } } YdbStatus: SCHEME_ERROR ConsumedRu: 1 }
2025-11-05T14:13:57.001349Z node 3 :KQP_COMPILE_ACTOR ERROR: Compilation failed, self: [3:7569244362557604321:2319], status: SCHEME_ERROR, issues: <main>: Error: Type annotation, code: 1030
    <main>:3:16: Error: At function: KiReadTable!
        <main>:3:16: Error: Cannot find table 'db.[/Root/PQ/Config/V2/Cluster]' because it does not exist or you do not have access permissions. Please check correctness of table path and user permissions., code: 2003
2025-11-05T14:13:57.003709Z node 3 :KQP_SESSION WARN: SessionId: ydb://session/3?node_id=3&id=NDRkMGNkYjgtZmYwYmZkOGEtMzJkYjAxMTAtZGZiMWUzM2U=, ActorId: [3:7569244362557604318:2317], ActorState: ExecuteState, TraceId: 01k9a5vz3b6b82r9yqe76ms5ds, ReplyQueryCompileError, status SCHEME_ERROR remove tx with tx_id: 
2025-11-05T14:13:57.004148Z node 3 :PERSQUEUE_CLUSTER_TRACKER ERROR: failed to list clusters: { Response { QueryIssues { message: "Type annotation" issue_code: 1030 severity: 1 issues { position { row: 3 column: 16 } message: "At function: KiReadTable!" end_position { row: 3 column: 16 } severity: 1 issues { position { row: 3 column: 16 } message: "Cannot find table \'db.[/Root/PQ/Config/V2/Cluster]\' because it does not exist or you do not have access permissions. Please check correctness of table path and user permissions." end_position { row: 3 column: 16 } issue_code: 2003 severity: 1 } } } TxMeta { } } YdbStatus: SCHEME_ERROR ConsumedRu: 1 }
2025-11-05T14:13:57.020768Z node 3 :FLAT_TX_SCHEMESHARD WARN: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpCreateTable, opId: 281474976715661:0, at schemeshard:  72057594046644480
2025-11-05T14:13:57.126489Z node 3 :FLAT_TX_SCHEMESHARD WARN: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpCreateTable, opId: 281474976715662:0, at schemeshard:  72057594046644480
2025-11-05T14:13:57.269758Z node 3 :FLAT_TX_SCHEMESHARD WARN: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpCreateTable, opId: 281474976715663:0, at schemeshard:  72057594046644480
=== Init DC: UPSERT INTO `/Root/PQ/Config/V2/Cluster` (name, balancer, local, enabled, weight) VALUES ("dc1", "localhost:20226", true, true, 1000), ("dc2", "dc2.logbroker.yandex.net", false, false, 1000);
2025-11-05T14:13:57.529607Z node 3 :KQP_EXECUTER ERROR: TxId: 281474976715664. Ctx: { TraceId: 01k9a5vzg40by4vgkvhwyf3vjk, Database: , DatabaseId: /Root, SessionId: ydb://session/3?node_id=3&id=MzhmYmI2ODMtZDcyMTU1NTYtMjgyMzc0NWItMTI2YTYzZDY=, CurrentExecutionId: , CustomerSuppliedId: , PoolId: }. Database not set, use /Root
=== CheckClustersList. Subcribe to ClusterTracker from [3:7569244366852572014:2914] 
2025-11-05T14:13:57.916639Z node 3 :METADATA_PROVIDER ERROR: fline=table_exists.cpp:59;actor=TTableExistsActor;event=timeout;self_id=[3:7569244345377734253:2149];send_to=[0:7307199536658146131:7762515];
2025-11-05T14:13:57.916713Z node 3 :METADATA_PROVIDER ERROR: fline=accessor_snapshot_base.cpp:69;action=cannot detect path existence;path=//Root/.metadata/initialization/migrations;error=timeout;
2025-11-05T14:13:57.943246Z node 4 :METADATA_PROVIDER ERROR: fline=table_exists.cpp:59;actor=TTableExistsActor;event=timeout;self_id=[4:7569244341998144109:2151];send_to=[0:7307199536658146131:7762515];
2025-11-05T14:13:57.943319Z node 4 :METADATA_PROVIDER ERROR: fline=accessor_snapshot_base.cpp:69;action=cannot detect path existence;path=//Root/.metadata/initialization/migrations;error=timeout;
=== CheckClustersList. Ok
2025-11-05T14:14:03.807294Z node 3 :FLAT_TX_SCHEMESHARD WARN: Operation part proposed ok, but propose itself is undo unsafe, suboperation type: ESchemeOpCreateTable, opId: 281474976715678:0, at schemeshard:  72057594046644480
waiting...
PQ Client: create topic: rt3.dc1--test-topic with 1 partitions
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic"
  }
}
2025-11-05T14:14:03.874642Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdCreateTopic {
    Topic: "rt3.dc1--test-topic"
    NumPartitions: 1
    Config {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      ReadRules: "user"
      ReadFromTimestampsMs: 0
      ConsumerFormatVersions: 0
      ConsumerCodecs {
      }
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      ReadRuleVersions: 0
    }
  }
}
CallPersQueueGRPC response:
Status: 129
ProxyErrorCode: 53
SchemeStatus: 1
FlatTxId {
  TxId: 281474976715679
  SchemeShardTabletId: 72057594046644480
  PathId: 9
}
ErrorCode: OK
AddTopic: rt3.dc1--test-topic
===Run query:``DECLARE $version as Int64; DECLARE $path AS Utf8; DECLARE $cluster as Utf8; UPSERT INTO `/Root/PQ/Config/V2/Topics` (path, dc) VALUES ($path, $cluster); UPSERT INTO `/Root/PQ/Config/V2/Versions` (name, version) VALUES ("Topics", $version);`` with topic = test-topic, dc = dc1
2025-11-05T14:14:03.899275Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][] pipe [3:7569244392622376410:3295] connected; active server actors: 1
2025-11-05T14:14:03.899587Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] updating configuration. Deleted partitions []. Added partitions [0]
2025-11-05T14:14:03.902379Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] Discovered subdomain [OwnerId: 72057594046644480, LocalPathId: 1] state, outOfSpace = 0 at RB 72075186224037893
2025-11-05T14:14:03.902563Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Handle TEvInterconnect::TEvNodeInfo
2025-11-05T14:14:03.902743Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Registered with mediator time cast
2025-11-05T14:14:03.903487Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] BALANCER INIT DONE for rt3.dc1--test-topic: (0, 72075186224037892) 
2025-11-05T14:14:03.903876Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] TEvClientConnected TabletId 72057594046644480, NodeId 3, Generation 2
2025-11-05T14:14:03.904751Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Transactions request. From tx_00000000000000000000, To tx_18446744073709551615
2025-11-05T14:14:03.905020Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] doesn't have tx info
2025-11-05T14:14:03.905047Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] PlanStep 0, PlanTxId 0, ExecStep 0, ExecTxId 0
2025-11-05T14:14:03.905063Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] no config, start with empty partitions and default config
2025-11-05T14:14:03.905078Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Txs.size=0, PlannedTxs.size=0
2025-11-05T14:14:03.905102Z node 4 :PERSQUEUE NOTICE: [PQ: 72075186224037892] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-11-05T14:14:03.905136Z node 4 :PERSQUEUE INFO: [PQ: 72075186224037892] doesn't have tx writes info
2025-11-05T14:14:03.905354Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] server connected, pipe [3:7569244392622376440:2447], now have 1 active actors on pipe
2025-11-05T14:14:03.905394Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] server connected, pipe [3:7569244392622376409:3294], now have 1 active actors on pipe
2025-11-05T14:14:03.905525Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] TEvClientConnected TabletId 72075186224037892, NodeId 4, Generation 1
2025-11-05T14:14:03.905950Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Handle TEvPersQueue::TEvProposeTransaction SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } TxId: 281474976715679 Config { TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic" Version: 0 LocalDC: true ReadRules: "user" ReadFromTimestampsMs: 0 ConsumerFormatVersions: 0 ConsumerCodecs { } Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } ReadRuleVersions: 0 TopicPath: "/Root/PQ/rt3.dc1--test-topic" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } } }
2025-11-05T14:14:03.906125Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] Propose TxId 281474976715679, WriteId (empty maybe)
2025-11-05T14:14:03.906163Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Try execute txs with state UNKNOWN
2025-11-05T14:14:03.906181Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679, State UNKNOWN
2025-11-05T14:14:03.906201Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] schedule TEvProposeTransactionResult(PREPARED)
2025-11-05T14:14:03.906229Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from UNKNOWN to PREPARING
2025-11-05T14:14:03.906273Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] write key for TxId 281474976715679
2025-11-05T14:14:03.906571Z node 4 :PQ_TX DEBUG: [TxId: 281474976715679] save tx TxId: 281474976715679 State: PREPARED MinStep: 1762352043907 MaxStep: 18446744073709551615 Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { }
2025-11-05T14:14:03.906702Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:03.910132Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:03.910176Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Try execute txs with state PREPARING
2025-11-05T14:14:03.910191Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679, State PREPARING
2025-11-05T14:14:03.910208Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from PREPARING to PREPARED
2025-11-05T14:14:03.920383Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Handle TEvPersQueue::TEvProposeTransactionAttach TabletId: 72075186224037892 TxId: 281474976715679
2025-11-05T14:14:03.923308Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] server connected, pipe [3:7569244392622376467:3340], now have 1 active actors on pipe
2025-11-05T14:14:03.923779Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Handle TEvTxProcessing::TEvPlanStep Transactions { TxId: 281474976715679 AckTo { RawX1: 0 RawX2: 0 } } Step: 1762352043970 MediatorID: 72057594046382081 TabletID: 72075186224037892
2025-11-05T14:14:03.923836Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Try execute txs with state PREPARED
2025-11-05T14:14:03.923854Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679, State PREPARED
2025-11-05T14:14:03.923881Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from PREPARED to PLANNING
2025-11-05T14:14:03.923903Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] PlanStep 1762352043970, PlanTxId 281474976715679
2025-11-05T14:14:03.923933Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] write key for TxId 281474976715679
2025-11-05T14:14:03.924290Z node 4 :PQ_TX DEBUG: [TxId: 281474976715679] save tx TxId: 281474976715679 State: PLANNED MinStep: 1762352043907 MaxStep: 18446744073709551615 Step: 1762352043970 Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { }
2025-11-05T14:14:03.924429Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:03.932701Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:03.932751Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Try execute txs with state PLANNING
2025-11-05T14:14:03.932770Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679, State PLANNING
2025-11-05T14:14:03.932791Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from PLANNING to PLANNED
2025-11-05T14:14:03.932818Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxQueue.size 1
2025-11-05T14:14:03.932842Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] New ExecStep 1762352043970, ExecTxId 281474976715679
2025-11-05T14:14:03.933791Z node 4 :PERSQUEUE DEBUG: [rt3.dc1--test-topic:0:Initializer] Start initializing step TInitConfigStep
2025-11-05T14:14:03.933923Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from PLANNED to CALCULATING
2025-11-05T14:14:03.933926Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateInit] HandleOnInit TEvPQ::TEvProposePartitionConfig
2025-11-05T14:14:03.934307Z node 4 :PERSQUEUE DEBUG: [rt3.dc1--test-topic:0:Initializer] Start initializing step TInitInternalFieldsStep
2025-11-05T14:14:03.934576Z node 4 :PERSQUEUE INFO: [PQ: 72075186224037892, Partition: 0, State: StateInit] bootstrapping 0 [4:7569244389242785218:2393]
2025-11-05T14:14:03.937735Z node 4 :PERSQUEUE DEBUG: [rt3.dc1--test-topic:0:Initializer] Initializing completed.
2025-11-05T14:14:03.937768Z node 4 :PERSQUEUE INFO: [PQ: 72075186224037892, Partition: 0, State: StateInit] init complete for topic 'rt3.dc1--test-topic' partition 0 generation 1 [4:7569244389242785218:2393]
2025-11-05T14:14:03.937797Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateInit] SYNC INIT topic rt3.dc1--test-topic partitition 0 so 0 endOffset 0 Head Offset 0 PartNo 0 PackedSize 0 count 0 nextOffset 0 batches 0
2025-11-05T14:14:03.939155Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Process pending events. Count 1
2025-11-05T14:14:03.939191Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:03.939279Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:03.939307Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Still pending for come commits
2025-11-05T14:14:03.939327Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:03.939327Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Handle TEvPQ::TEvProposePartitionConfigResult Step 1762352043970, TxId 281474976715679, Partition 0
2025-11-05T14:14:03.939344Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:03.939348Z node 4 :PQ_TX DEBUG: [TxId: 281474976715679] Handle TEvProposePartitionConfigResult
2025-11-05T14:14:03.939360Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Still pending for come commits
2025-11-05T14:14:03.939371Z node 4 :PQ_TX DEBUG: [TxId: 281474976715679] Partition responses 1/1
2025-11-05T14:14:03.939388Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Try execute txs with state CALCULATING
2025-11-05T14:14:03.939407Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-11-05T14:14:03.939413Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679, State CALCULATING
2025-11-05T14:14:03.939433Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679 State CALCULATING FrontTxId 281474976715679
2025-11-05T14:14:03.939450Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Received 1, Expected 1
2025-11-05T14:14:03.939472Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from CALCULATING to CALCULATED
2025-11-05T14:14:03.939533Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] write key for TxId 281474976715679
2025-11-05T14:14:03.939903Z node 4 :PQ_TX DEBUG: [TxId: 281474976715679] save tx TxId: 281474976715679 State: CALCULATED MinStep: 1762352043907 MaxStep: 18446744073709551615 Step: 1762352043970 Predicate: true Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { Partition { PartitionId: 0 } }
2025-11-05T14:14:03.940003Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:03.946428Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:03.946485Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Try execute txs with state CALCULATED
2025-11-05T14:14:03.946504Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679, State CALCULATED
2025-11-05T14:14:03.946546Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679 State CALCULATED FrontTxId 281474976715679
2025-11-05T14:14:03.946566Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from CALCULATED to WAIT_RS
2025-11-05T14:14:03.946623Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] Send TEvTxProcessing::TEvReadSet to 0 receivers. Wait TEvTxProcessing::TEvReadSet from 0 senders.
2025-11-05T14:14:03.946649Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] HaveParticipantsDecision 1
2025-11-05T14:14:03.946739Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from WAIT_RS to EXECUTING
2025-11-05T14:14:03.946756Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Received 0, Expected 1
2025-11-05T14:14:03.946778Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Handle TEvPQ::TEvTxCommit Step 1762352043970, TxId 281474976715679
2025-11-05T14:14:03.946804Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:03.946822Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:03.946963Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:03.947161Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:03.950083Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] Handle TEvPQ::TEvTxCommitDone Step 1762352043970, TxId 281474976715679, Partition 0
2025-11-05T14:14:03.950114Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Try execute txs with state EXECUTING
2025-11-05T14:14:03.950130Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679, State EXECUTING
2025-11-05T14:14:03.950147Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679 State EXECUTING FrontTxId 281474976715679
2025-11-05T14:14:03.950163Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Received 1, Expected 1
2025-11-05T14:14:03.950188Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId: 281474976715679 send TEvPersQueue::TEvProposeTransactionResult(COMPLETE)
2025-11-05T14:14:03.950227Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] complete TxId 281474976715679
2025-11-05T14:14:03.950552Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Apply new config PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false }
2025-11-05T14:14:03.950677Z node 4 :PERSQUEUE NOTICE: [PQ: 72075186224037892] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-11-05T14:14:03.950701Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:03.950735Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] delete partitions for TxId 281474976715679
2025-11-05T14:14:03.950739Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:03.950754Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from EXECUTING to EXECUTED
2025-11-05T14:14:03.950759Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:03.950781Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] write key for TxId 281474976715679
2025-11-05T14:14:03.951076Z node 4 :PQ_TX DEBUG: [TxId: 281474976715679] save tx TxId: 281474976715679 State: EXECUTED MinStep: 1762352043907 MaxStep: 18446744073709551615 Step: 1762352043970 Predicate: true Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { Partition { PartitionId: 0 } }
2025-11-05T14:14:03.951253Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:03.955650Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:03.955681Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Try execute txs with state EXECUTED
2025-11-05T14:14:03.955699Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679, State EXECUTED
2025-11-05T14:14:03.955715Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679 State EXECUTED FrontTxId 281474976715679
2025-11-05T14:14:03.955732Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TPersQueue::SendEvReadSetAckToSenders
2025-11-05T14:14:03.955747Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from EXECUTED to WAIT_RS_ACKS
2025-11-05T14:14:03.955786Z node 4 :PQ_TX DEBUG: [TxId: 281474976715679] PredicateAcks: 0/0
2025-11-05T14:14:03.955798Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] HaveAllRecipientsReceive 1, AllSupportivePartitionsHaveBeenDeleted 1
2025-11-05T14:14:03.955810Z node 4 :PQ_TX DEBUG: [TxId: 281474976715679] PredicateAcks: 0/0
2025-11-05T14:14:03.955824Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] add an TxId 281474976715679 to the list for deletion
2025-11-05T14:14:03.955840Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] TxId 281474976715679 moved from WAIT_RS_ACKS to DELETING
2025-11-05T14:14:03.955863Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] delete key for TxId 281474976715679
2025-11-05T14:14:03.955906Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:03.957831Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:03.957859Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] Try execute txs with state DELETING
2025-11-05T14:14:03.957874Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037892] TxId 281474976715679, State DELETING
2025-11-05T14:14:03.957902Z node 4 :PQ_TX INFO: [PQ: 72075186224037892] delete TxId 281474976715679
2025-11-05T14:14:04.044794Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.044853Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:04.047576Z node 3 :KQP_EXECUTER ERROR: TxId: 281474976715680. Ctx: { TraceId: 01k9a5w5w59gtkv7dfy89ss9w0, Database: , DatabaseId: /Root, SessionId: ydb://session/3?node_id=3&id=MmExYzJlOGMtYjE4MDMxOTMtNGZlMWIzN2ItY2ZkYWIxYTY=, CurrentExecutionId: , CustomerSuppliedId: , PoolId: }. Database not set, use /Root
===Query complete
TClient::Ls request: /Root/PQ/rt3.dc1--test-topic
TClient::Ls response: Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic"
    PathId: 9
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976715679
    CreateStep: 1762352043970
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
    BalancerTabletID: 72075186224037893
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic"
    PathId: 9
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSi...
(TRUNCATED)
GetTopicVersionFromPath:  record Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic"
    PathId: 9
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976715679
    CreateStep: 1762352043970
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
    BalancerTabletID: 72075186224037893
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic"
    PathId: 9
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      ReadRules: "user"
      ReadFromTimestampsMs: 0
      ConsumerFormatVersions: 0
      ConsumerCodecs {
      }
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      ReadRuleVersions: 0
      YdbDatabasePath: "/Root"
    }
    Partitions {
      PartitionId: 0
      TabletId: 72075186224037892
      Status: Active
    }
    AlterVersion: 1
    BalancerTabletID: 72075186224037893
    NextPartitionId: 1
  }
  DomainDescription {
    SchemeShardId_Depricated: 72057594046644480
    PathId_Depricated: 1
    ProcessingParams {
      Version: 1
      PlanResolution: 7
      Coordinators: 72057594046316545
      TimeCastBucketsPerMediator: 2
      Mediators: 72057594046382081
    }
    DomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    PathsInside: 8
    PathsLimit: 10000
    ShardsInside: 6
    ShardsLimit: 200000
    ResourcesDomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    DiskSpaceUsage {
      Tables {
        TotalSize: 0
        DataSize: 0
        IndexSize: 0
      }
      Topics {
        ReserveSize: 0
        AccountSize: 0
        DataSize: 0
        UsedReserveSize: 0
      }
    }
    PQPartitionsInside: 1
    PQPartitionsLimit: 1000000
  }
}
Path: "/Root/PQ/rt3.dc1--test-topic"
 name rt3.dc1--test-topic version1
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic"
  }
}
2025-11-05T14:14:04.084357Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
2025-11-05T14:14:04.145291Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.145335Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:04.246041Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.246080Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:04.346381Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.346420Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:04.446749Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.446792Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:04.547086Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.547131Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic"
  }
}
2025-11-05T14:14:04.586957Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
2025-11-05T14:14:04.647491Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.647552Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:04.747818Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.747861Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:04.848168Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.848212Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:04.948454Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:04.948488Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.048787Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.048815Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic"
  }
}
2025-11-05T14:14:05.091553Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 1
ErrorCode: OK
MetaResponse {
  CmdGetTopicMetadataResult {
    TopicInfo {
      Topic: "rt3.dc1--test-topic"
      NumPartitions: 1
      Config {
        PartitionConfig {
          LifetimeSeconds: 86400
          LowWatermark: 8388608
          SourceIdLifetimeSeconds: 86400
          WriteSpeedInBytesPerSecond: 20000000
          BurstSize: 20000000
          SourceIdMaxCounts: 6000000
        }
        Version: 1
        LocalDC: true
        Codecs {
          Ids: 0
          Ids: 1
          Ids: 2
          Codecs: "raw"
          Codecs: "gzip"
          Codecs: "lzop"
        }
        TopicPath: "/Root/PQ/rt3.dc1--test-topic"
        YdbDatabasePath: "/Root"
        Consumers {
          Name: "user"
          ReadFromTimestampsMs: 0
          FormatVersion: 0
          Codec {
          }
          Version: 0
          Important: false
        }
      }
      ErrorCode: OK
    }
  }
}
=== Topic created, have version: 1
2025-11-05T14:14:05.102877Z :DEBUG: [] MessageGroupId [src] SessionId [] Write session: try to update token
2025-11-05T14:14:05.103595Z :INFO: [] MessageGroupId [src] SessionId [] Write session: Do CDS request
2025-11-05T14:14:05.103634Z :INFO: [] MessageGroupId [src] SessionId [] Start write session. Will connect to endpoint: localhost:20226
2025-11-05T14:14:05.111951Z :DEBUG: [] MessageGroupId [src] SessionId [] Write session: send init request: init_request { topic: "test-topic" message_group_id: "src" }
2025-11-05T14:14:05.113298Z node 3 :PQ_WRITE_PROXY DEBUG: new grpc connection
2025-11-05T14:14:05.113334Z node 3 :PQ_WRITE_PROXY DEBUG: new session created cookie 1
2025-11-05T14:14:05.113668Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 1 sessionId:  grpc read done: success: 1 data: init_request { topic: "test-topic" message_group_id: "src" }
2025-11-05T14:14:05.113841Z node 3 :PQ_WRITE_PROXY INFO: session request cookie: 1 topic: "test-topic" message_group_id: "src" from ipv6:[::1]:55076
2025-11-05T14:14:05.113870Z node 3 :PQ_WRITE_PROXY INFO: write session:  cookie=1 sessionId= userAgent="pqv1 server" ip=ipv6:[::1]:55076 proto=v1  topic=test-topic durationSec=0
2025-11-05T14:14:05.113891Z node 3 :PQ_WRITE_PROXY INFO: init check schema
2025-11-05T14:14:05.115541Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 1 sessionId:  describe result for acl check
2025-11-05T14:14:05.115661Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper SelectQuery: --!syntax_v1
DECLARE $Hash AS Uint32; DECLARE $Topic AS Utf8; DECLARE $SourceId AS Utf8;
SELECT Partition, CreateTime, AccessTime, SeqNo FROM `/Root/PQ/SourceIdMeta2` WHERE Hash == $Hash AND Topic == $Topic AND SourceId == $SourceId;
2025-11-05T14:14:05.115675Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper UpdateQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;DECLARE $SeqNo AS Uint64;
UPSERT INTO `/Root/PQ/SourceIdMeta2` (Hash, Topic, SourceId, CreateTime, AccessTime, Partition, SeqNo) VALUES ($Hash, $Topic, $SourceId, $CreateTime, $AccessTime, $Partition, $SeqNo);
2025-11-05T14:14:05.115684Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper UpdateAccessTimeQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;
UPDATE `/Root/PQ/SourceIdMeta2` SET AccessTime = $AccessTime WHERE Hash = $Hash AND Topic = $Topic AND SourceId = $SourceId AND Partition = $Partition;
2025-11-05T14:14:05.115701Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244401212311260:2485] (SourceId=src, PreferedPartition=(NULL)) StartKqpSession
2025-11-05T14:14:05.118172Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244401212311260:2485] (SourceId=src, PreferedPartition=(NULL)) Select from the table
2025-11-05T14:14:05.149190Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.149229Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.252072Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.252117Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.264192Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244401212311260:2485] (SourceId=src, PreferedPartition=(NULL)) RequestPQRB
2025-11-05T14:14:05.264918Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] pipe [3:7569244401212311298:2485] connected; active server actors: 1
2025-11-05T14:14:05.264967Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244401212311260:2485] (SourceId=src, PreferedPartition=(NULL)) Received partition 0 from PQRB for SourceId=src
2025-11-05T14:14:05.264983Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244401212311260:2485] (SourceId=src, PreferedPartition=(NULL)) Update the table
2025-11-05T14:14:05.265565Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] pipe [3:7569244401212311298:2485] disconnected; active server actors: 1
2025-11-05T14:14:05.265597Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] pipe [3:7569244401212311298:2485] disconnected no session
2025-11-05T14:14:05.352452Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.352492Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.392612Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244401212311260:2485] (SourceId=src, PreferedPartition=(NULL)) HandleUpdate PartitionPersisted=0 Status=SUCCESS
2025-11-05T14:14:05.392652Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244401212311260:2485] (SourceId=src, PreferedPartition=(NULL)) ReplyResult: Partition=0, SeqNo=(NULL)
2025-11-05T14:14:05.392688Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244401212311260:2485] (SourceId=src, PreferedPartition=(NULL)) Start idle
2025-11-05T14:14:05.392725Z node 3 :PQ_WRITE_PROXY DEBUG: ProceedPartition. session cookie: 1 sessionId:  partition: 0 expectedGeneration: (NULL)
2025-11-05T14:14:05.393686Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) TEvClientConnected Status OK, TabletId: 72075186224037892, NodeId 4, Generation: 1
2025-11-05T14:14:05.396794Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] server connected, pipe [3:7569244401212311321:2485], now have 1 active actors on pipe
2025-11-05T14:14:05.396896Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:05.396927Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:05.397050Z node 4 :PERSQUEUE INFO: new Cookie src|e5612eed-73869a85-91388714-f88d11ec_0 generated for partition 0 topic 'rt3.dc1--test-topic' owner src
2025-11-05T14:14:05.397122Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.397156Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.397195Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.397253Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::ReplyOwnerOk. Partition: 0
2025-11-05T14:14:05.397324Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:05.397986Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:05.398015Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:05.398076Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:05.398412Z node 3 :PQ_WRITE_PROXY INFO: session inited cookie: 1 partition: 0 MaxSeqNo: 0 sessionId: src|e5612eed-73869a85-91388714-f88d11ec_0
2025-11-05T14:14:05.400212Z :INFO: [] MessageGroupId [src] SessionId [] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1762352045400 BytesWritten: 0 MessagesWritten: 0 BytesWrittenCompressed: 0 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-11-05T14:14:05.400332Z :INFO: [] MessageGroupId [src] SessionId [] Write session established. Init response: session_id: "src|e5612eed-73869a85-91388714-f88d11ec_0" topic: "test-topic" cluster: "dc1" supported_codecs: CODEC_RAW supported_codecs: CODEC_GZIP supported_codecs: CODEC_LZOP
2025-11-05T14:14:05.400650Z :INFO: [] MessageGroupId [src] SessionId [src|e5612eed-73869a85-91388714-f88d11ec_0] Write session: close. Timeout = 0 ms
2025-11-05T14:14:05.400711Z :INFO: [] MessageGroupId [src] SessionId [src|e5612eed-73869a85-91388714-f88d11ec_0] Write session will now close
2025-11-05T14:14:05.400768Z :DEBUG: [] MessageGroupId [src] SessionId [src|e5612eed-73869a85-91388714-f88d11ec_0] Write session: aborting
2025-11-05T14:14:05.405838Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 1 sessionId: src|e5612eed-73869a85-91388714-f88d11ec_0 grpc read done: success: 0 data: 
2025-11-05T14:14:05.405864Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 1 sessionId: src|e5612eed-73869a85-91388714-f88d11ec_0 grpc read failed
2025-11-05T14:14:05.405893Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 1 sessionId: src|e5612eed-73869a85-91388714-f88d11ec_0 grpc closed
2025-11-05T14:14:05.405910Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 1 sessionId: src|e5612eed-73869a85-91388714-f88d11ec_0 is DEAD
2025-11-05T14:14:05.406588Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::TEvents::TEvPoison
2025-11-05T14:14:05.407744Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] server disconnected, pipe [3:7569244401212311321:2485] destroyed
2025-11-05T14:14:05.407808Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::DropOwner.
2025-11-05T14:14:05.407845Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.407867Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.408182Z :INFO: [] MessageGroupId [src] SessionId [src|e5612eed-73869a85-91388714-f88d11ec_0] Write session: gracefully shut down, all writes complete
2025-11-05T14:14:05.408303Z :DEBUG: [] MessageGroupId [src] SessionId [src|e5612eed-73869a85-91388714-f88d11ec_0] Write session is aborting and will not restart
2025-11-05T14:14:05.408459Z :DEBUG: [] MessageGroupId [src] SessionId [src|e5612eed-73869a85-91388714-f88d11ec_0] Write session: destroy
PQ Client: create topic: rt3.dc1--test-topic-mirrored-from-dc2 with 1 partitions
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc2"
  }
}
2025-11-05T14:14:05.422154Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic-mirrored-from-dc2, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdCreateTopic {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc2"
    NumPartitions: 1
    Config {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      ReadRules: "user"
      ReadFromTimestampsMs: 0
      ConsumerFormatVersions: 0
      ConsumerCodecs {
      }
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      ReadRuleVersions: 0
    }
  }
}
CallPersQueueGRPC response:
Status: 129
ProxyErrorCode: 53
SchemeStatus: 1
FlatTxId {
  TxId: 281474976715687
  SchemeShardTabletId: 72057594046644480
  PathId: 10
}
ErrorCode: OK
AddTopic: rt3.dc1--test-topic-mirrored-from-dc2
===Run query:``DECLARE $version as Int64; DECLARE $path AS Utf8; DECLARE $cluster as Utf8; UPSERT INTO `/Root/PQ/Config/V2/Topics` (path, dc) VALUES ($path, $cluster); UPSERT INTO `/Root/PQ/Config/V2/Versions` (name, version) VALUES ("Topics", $version);`` with topic = test-topic-mirrored-from-dc2, dc = dc1
2025-11-05T14:14:05.448840Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037895][] pipe [3:7569244401212311384:3530] connected; active server actors: 1
2025-11-05T14:14:05.449534Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] updating configuration. Deleted partitions []. Added partitions [0]
2025-11-05T14:14:05.450290Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] Discovered subdomain [OwnerId: 72057594046644480, LocalPathId: 1] state, outOfSpace = 0 at RB 72075186224037895
2025-11-05T14:14:05.452810Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] BALANCER INIT DONE for rt3.dc1--test-topic-mirrored-from-dc2: (0, 72075186224037894) 
2025-11-05T14:14:05.453553Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] TEvClientConnected TabletId 72057594046644480, NodeId 3, Generation 2
2025-11-05T14:14:05.454060Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.454100Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.455445Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Handle TEvInterconnect::TEvNodeInfo
2025-11-05T14:14:05.458138Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Registered with mediator time cast
2025-11-05T14:14:05.459540Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Transactions request. From tx_00000000000000000000, To tx_18446744073709551615
2025-11-05T14:14:05.459799Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] doesn't have tx info
2025-11-05T14:14:05.459818Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] PlanStep 0, PlanTxId 0, ExecStep 0, ExecTxId 0
2025-11-05T14:14:05.459832Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] no config, start with empty partitions and default config
2025-11-05T14:14:05.459865Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Txs.size=0, PlannedTxs.size=0
2025-11-05T14:14:05.459889Z node 4 :PERSQUEUE NOTICE: [PQ: 72075186224037894] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-11-05T14:14:05.459932Z node 4 :PERSQUEUE INFO: [PQ: 72075186224037894] doesn't have tx writes info
2025-11-05T14:14:05.461681Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] server connected, pipe [3:7569244401212311380:3529], now have 1 active actors on pipe
2025-11-05T14:14:05.461710Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] server connected, pipe [3:7569244401212311399:2496], now have 1 active actors on pipe
2025-11-05T14:14:05.462082Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] TEvClientConnected TabletId 72075186224037894, NodeId 4, Generation 1
2025-11-05T14:14:05.463075Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Handle TEvPersQueue::TEvProposeTransaction SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } TxId: 281474976715687 Config { TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc2" Version: 0 LocalDC: true ReadRules: "user" ReadFromTimestampsMs: 0 ConsumerFormatVersions: 0 ConsumerCodecs { } Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } ReadRuleVersions: 0 TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } } }
2025-11-05T14:14:05.463202Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] Propose TxId 281474976715687, WriteId (empty maybe)
2025-11-05T14:14:05.463231Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Try execute txs with state UNKNOWN
2025-11-05T14:14:05.463250Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687, State UNKNOWN
2025-11-05T14:14:05.463278Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] schedule TEvProposeTransactionResult(PREPARED)
2025-11-05T14:14:05.463302Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from UNKNOWN to PREPARING
2025-11-05T14:14:05.463342Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] write key for TxId 281474976715687
2025-11-05T14:14:05.463601Z node 4 :PQ_TX DEBUG: [TxId: 281474976715687] save tx TxId: 281474976715687 State: PREPARED MinStep: 1762352045307 MaxStep: 18446744073709551615 Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc2" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { }
2025-11-05T14:14:05.463691Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:05.468385Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:05.468423Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Try execute txs with state PREPARING
2025-11-05T14:14:05.468447Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687, State PREPARING
2025-11-05T14:14:05.468467Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from PREPARING to PREPARED
2025-11-05T14:14:05.474899Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Handle TEvPersQueue::TEvProposeTransactionAttach TabletId: 72075186224037894 TxId: 281474976715687
2025-11-05T14:14:05.479862Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] server connected, pipe [3:7569244401212311431:3569], now have 1 active actors on pipe
2025-11-05T14:14:05.480433Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Handle TEvTxProcessing::TEvPlanStep Transactions { TxId: 281474976715687 AckTo { RawX1: 0 RawX2: 0 } } Step: 1762352045517 MediatorID: 72057594046382081 TabletID: 72075186224037894
2025-11-05T14:14:05.480474Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Try execute txs with state PREPARED
2025-11-05T14:14:05.480492Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687, State PREPARED
2025-11-05T14:14:05.480515Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from PREPARED to PLANNING
2025-11-05T14:14:05.480535Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] PlanStep 1762352045517, PlanTxId 281474976715687
2025-11-05T14:14:05.480557Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] write key for TxId 281474976715687
2025-11-05T14:14:05.480901Z node 4 :PQ_TX DEBUG: [TxId: 281474976715687] save tx TxId: 281474976715687 State: PLANNED MinStep: 1762352045307 MaxStep: 18446744073709551615 Step: 1762352045517 Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc2" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { }
2025-11-05T14:14:05.480991Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:05.485031Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:05.485096Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Try execute txs with state PLANNING
2025-11-05T14:14:05.485114Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687, State PLANNING
2025-11-05T14:14:05.485137Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from PLANNING to PLANNED
2025-11-05T14:14:05.485160Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxQueue.size 1
2025-11-05T14:14:05.485179Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] New ExecStep 1762352045517, ExecTxId 281474976715687
2025-11-05T14:14:05.485832Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from PLANNED to CALCULATING
2025-11-05T14:14:05.485899Z node 4 :PERSQUEUE DEBUG: [rt3.dc1--test-topic-mirrored-from-dc2:0:Initializer] Start initializing step TInitConfigStep
2025-11-05T14:14:05.485950Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateInit] HandleOnInit TEvPQ::TEvProposePartitionConfig
2025-11-05T14:14:05.486227Z node 4 :PERSQUEUE DEBUG: [rt3.dc1--test-topic-mirrored-from-dc2:0:Initializer] Start initializing step TInitInternalFieldsStep
2025-11-05T14:14:05.486420Z node 4 :PERSQUEUE INFO: [PQ: 72075186224037894, Partition: 0, State: StateInit] bootstrapping 0 [4:7569244397832720006:2422]
2025-11-05T14:14:05.489748Z node 4 :PERSQUEUE DEBUG: [rt3.dc1--test-topic-mirrored-from-dc2:0:Initializer] Initializing completed.
2025-11-05T14:14:05.489786Z node 4 :PERSQUEUE INFO: [PQ: 72075186224037894, Partition: 0, State: StateInit] init complete for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 generation 1 [4:7569244397832720006:2422]
2025-11-05T14:14:05.489817Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateInit] SYNC INIT topic rt3.dc1--test-topic-mirrored-from-dc2 partitition 0 so 0 endOffset 0 Head Offset 0 PartNo 0 PackedSize 0 count 0 nextOffset 0 batches 0
2025-11-05T14:14:05.491101Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Process pending events. Count 1
2025-11-05T14:14:05.491135Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.491216Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.491245Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Still pending for come commits
2025-11-05T14:14:05.491264Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.491290Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.491310Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Still pending for come commits
2025-11-05T14:14:05.491351Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-11-05T14:14:05.491599Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Handle TEvPQ::TEvProposePartitionConfigResult Step 1762352045517, TxId 281474976715687, Partition 0
2025-11-05T14:14:05.491619Z node 4 :PQ_TX DEBUG: [TxId: 281474976715687] Handle TEvProposePartitionConfigResult
2025-11-05T14:14:05.491636Z node 4 :PQ_TX DEBUG: [TxId: 281474976715687] Partition responses 1/1
2025-11-05T14:14:05.491652Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Try execute txs with state CALCULATING
2025-11-05T14:14:05.491690Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687, State CALCULATING
2025-11-05T14:14:05.491711Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687 State CALCULATING FrontTxId 281474976715687
2025-11-05T14:14:05.491726Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Received 1, Expected 1
2025-11-05T14:14:05.491746Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from CALCULATING to CALCULATED
2025-11-05T14:14:05.491772Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] write key for TxId 281474976715687
2025-11-05T14:14:05.492122Z node 4 :PQ_TX DEBUG: [TxId: 281474976715687] save tx TxId: 281474976715687 State: CALCULATED MinStep: 1762352045307 MaxStep: 18446744073709551615 Step: 1762352045517 Predicate: true Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc2" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { Partition { PartitionId: 0 } }
2025-11-05T14:14:05.492221Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:05.498632Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:05.498661Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Try execute txs with state CALCULATED
2025-11-05T14:14:05.498707Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687, State CALCULATED
2025-11-05T14:14:05.498742Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687 State CALCULATED FrontTxId 281474976715687
2025-11-05T14:14:05.498762Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from CALCULATED to WAIT_RS
2025-11-05T14:14:05.498810Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] Send TEvTxProcessing::TEvReadSet to 0 receivers. Wait TEvTxProcessing::TEvReadSet from 0 senders.
2025-11-05T14:14:05.498833Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] HaveParticipantsDecision 1
2025-11-05T14:14:05.498888Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from WAIT_RS to EXECUTING
2025-11-05T14:14:05.498915Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Received 0, Expected 1
2025-11-05T14:14:05.498957Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Handle TEvPQ::TEvTxCommit Step 1762352045517, TxId 281474976715687
2025-11-05T14:14:05.498985Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.499004Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.499146Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.499325Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:05.502254Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] Handle TEvPQ::TEvTxCommitDone Step 1762352045517, TxId 281474976715687, Partition 0
2025-11-05T14:14:05.502284Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:05.502290Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Try execute txs with state EXECUTING
2025-11-05T14:14:05.502307Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687, State EXECUTING
2025-11-05T14:14:05.502320Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.502325Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687 State EXECUTING FrontTxId 281474976715687
2025-11-05T14:14:05.502341Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.502342Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Received 1, Expected 1
2025-11-05T14:14:05.502371Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId: 281474976715687 send TEvPersQueue::TEvProposeTransactionResult(COMPLETE)
2025-11-05T14:14:05.502394Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] complete TxId 281474976715687
2025-11-05T14:14:05.502719Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Apply new config PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc2" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false }
2025-11-05T14:14:05.502860Z node 4 :PERSQUEUE NOTICE: [PQ: 72075186224037894] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-11-05T14:14:05.502916Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] delete partitions for TxId 281474976715687
2025-11-05T14:14:05.502943Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from EXECUTING to EXECUTED
2025-11-05T14:14:05.502974Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] write key for TxId 281474976715687
2025-11-05T14:14:05.503289Z node 4 :PQ_TX DEBUG: [TxId: 281474976715687] save tx TxId: 281474976715687 State: EXECUTED MinStep: 1762352045307 MaxStep: 18446744073709551615 Step: 1762352045517 Predicate: true Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc2" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { Partition { PartitionId: 0 } }
2025-11-05T14:14:05.503475Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:05.509880Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:05.509916Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Try execute txs with state EXECUTED
2025-11-05T14:14:05.509933Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687, State EXECUTED
2025-11-05T14:14:05.509956Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687 State EXECUTED FrontTxId 281474976715687
2025-11-05T14:14:05.509974Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TPersQueue::SendEvReadSetAckToSenders
2025-11-05T14:14:05.509992Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from EXECUTED to WAIT_RS_ACKS
2025-11-05T14:14:05.510024Z node 4 :PQ_TX DEBUG: [TxId: 281474976715687] PredicateAcks: 0/0
2025-11-05T14:14:05.510035Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] HaveAllRecipientsReceive 1, AllSupportivePartitionsHaveBeenDeleted 1
2025-11-05T14:14:05.510046Z node 4 :PQ_TX DEBUG: [TxId: 281474976715687] PredicateAcks: 0/0
2025-11-05T14:14:05.510061Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] add an TxId 281474976715687 to the list for deletion
2025-11-05T14:14:05.510086Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] TxId 281474976715687 moved from WAIT_RS_ACKS to DELETING
2025-11-05T14:14:05.510114Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] delete key for TxId 281474976715687
2025-11-05T14:14:05.510181Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:05.512805Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:05.512841Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] Try execute txs with state DELETING
2025-11-05T14:14:05.512884Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037894] TxId 281474976715687, State DELETING
2025-11-05T14:14:05.512910Z node 4 :PQ_TX INFO: [PQ: 72075186224037894] delete TxId 281474976715687
2025-11-05T14:14:05.554230Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.554269Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.591518Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.591557Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.637220Z node 3 :KQP_EXECUTER ERROR: TxId: 281474976715688. Ctx: { TraceId: 01k9a5w7d2f99d3ggahyek1wja, Database: , DatabaseId: /Root, SessionId: ydb://session/3?node_id=3&id=MzczNTZiYzYtZjExZDA1OGEtYmZiN2NkMmQtN2Y2MDZiMDQ=, CurrentExecutionId: , CustomerSuppliedId: , PoolId: }. Database not set, use /Root
2025-11-05T14:14:05.654546Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.654583Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
===Query complete
TClient::Ls request: /Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2
TClient::Ls response: Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic-mirrored-from-dc2"
    PathId: 10
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976715687
    CreateStep: 1762352045517
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
    BalancerTabletID: 72075186224037895
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic-mirrored-from-dc2"
    PathId: 10
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBy...
(TRUNCATED)
GetTopicVersionFromPath:  record Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic-mirrored-from-dc2"
    PathId: 10
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976715687
    CreateStep: 1762352045517
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
    BalancerTabletID: 72075186224037895
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic-mirrored-from-dc2"
    PathId: 10
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      ReadRules: "user"
      ReadFromTimestampsMs: 0
      ConsumerFormatVersions: 0
      ConsumerCodecs {
      }
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      ReadRuleVersions: 0
      YdbDatabasePath: "/Root"
    }
    Partitions {
      PartitionId: 0
      TabletId: 72075186224037894
      Status: Active
    }
    AlterVersion: 1
    BalancerTabletID: 72075186224037895
    NextPartitionId: 1
  }
  DomainDescription {
    SchemeShardId_Depricated: 72057594046644480
    PathId_Depricated: 1
    ProcessingParams {
      Version: 1
      PlanResolution: 7
      Coordinators: 72057594046316545
      TimeCastBucketsPerMediator: 2
      Mediators: 72057594046382081
    }
    DomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    PathsInside: 9
    PathsLimit: 10000
    ShardsInside: 8
    ShardsLimit: 200000
    ResourcesDomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    DiskSpaceUsage {
      Tables {
        TotalSize: 0
        DataSize: 0
        IndexSize: 0
      }
      Topics {
        ReserveSize: 0
        AccountSize: 0
        DataSize: 0
        UsedReserveSize: 0
      }
    }
    PQPartitionsInside: 2
    PQPartitionsLimit: 1000000
  }
}
Path: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2"
 name rt3.dc1--test-topic-mirrored-from-dc2 version1
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc2"
  }
}
2025-11-05T14:14:05.670803Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic-mirrored-from-dc2, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
2025-11-05T14:14:05.692059Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.692099Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.754938Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.754984Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.792313Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.792360Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.856570Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.856617Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.892902Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.892952Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.956604Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.956634Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:05.993017Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:05.993073Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.060854Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.060902Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.093469Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.093513Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.160861Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.160902Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc2"
  }
}
2025-11-05T14:14:06.179224Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 1
ErrorCode: OK
MetaResponse {
  CmdGetTopicMetadataResult {
    TopicInfo {
      Topic: "rt3.dc1--test-topic-mirrored-from-dc2"
      NumPartitions: 1
      Config {
        PartitionConfig {
          LifetimeSeconds: 86400
          LowWatermark: 8388608
          SourceIdLifetimeSeconds: 86400
          WriteSpeedInBytesPerSecond: 20000000
          BurstSize: 20000000
          SourceIdMaxCounts: 6000000
        }
        Version: 1
        LocalDC: true
        Codecs {
          Ids: 0
          Ids: 1
          Ids: 2
          Codecs: "raw"
          Codecs: "gzip"
          Codecs: "lzop"
        }
        TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2"
        YdbDatabasePath: "/Root"
        Consumers {
          Name: "user"
          ReadFromTimestampsMs: 0
          FormatVersion: 0
          Codec {
          }
          Version: 0
          Important: false
        }
      }
      ErrorCode: OK
    }
  }
}
=== Topic created, have version: 1
PQ Client: create topic: rt3.dc1--test-topic-mirrored-from-dc3 with 1 partitions
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc3"
  }
}
2025-11-05T14:14:06.183467Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic-mirrored-from-dc3, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdCreateTopic {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc3"
    NumPartitions: 1
    Config {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      ReadRules: "user"
      ReadFromTimestampsMs: 0
      ConsumerFormatVersions: 0
      ConsumerCodecs {
      }
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      ReadRuleVersions: 0
    }
  }
}
CallPersQueueGRPC response:
Status: 129
ProxyErrorCode: 53
SchemeStatus: 1
FlatTxId {
  TxId: 281474976715692
  SchemeShardTabletId: 72057594046644480
  PathId: 11
}
ErrorCode: OK
AddTopic: rt3.dc1--test-topic-mirrored-from-dc3
===Run query:``DECLARE $version as Int64; DECLARE $path AS Utf8; DECLARE $cluster as Utf8; UPSERT INTO `/Root/PQ/Config/V2/Topics` (path, dc) VALUES ($path, $cluster); UPSERT INTO `/Root/PQ/Config/V2/Versions` (name, version) VALUES ("Topics", $version);`` with topic = test-topic-mirrored-from-dc3, dc = dc1
2025-11-05T14:14:06.196530Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.196558Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.211044Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037897][] pipe [3:7569244405507278937:3704] connected; active server actors: 1
2025-11-05T14:14:06.211251Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] updating configuration. Deleted partitions []. Added partitions [0]
2025-11-05T14:14:06.211965Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] Discovered subdomain [OwnerId: 72057594046644480, LocalPathId: 1] state, outOfSpace = 0 at RB 72075186224037897
2025-11-05T14:14:06.214229Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Handle TEvInterconnect::TEvNodeInfo
2025-11-05T14:14:06.214540Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Registered with mediator time cast
2025-11-05T14:14:06.214021Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] BALANCER INIT DONE for rt3.dc1--test-topic-mirrored-from-dc3: (0, 72075186224037896) 
2025-11-05T14:14:06.215570Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] TEvClientConnected TabletId 72057594046644480, NodeId 3, Generation 2
2025-11-05T14:14:06.216578Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Transactions request. From tx_00000000000000000000, To tx_18446744073709551615
2025-11-05T14:14:06.216793Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] doesn't have tx info
2025-11-05T14:14:06.216813Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] PlanStep 0, PlanTxId 0, ExecStep 0, ExecTxId 0
2025-11-05T14:14:06.216827Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] no config, start with empty partitions and default config
2025-11-05T14:14:06.216843Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Txs.size=0, PlannedTxs.size=0
2025-11-05T14:14:06.216866Z node 4 :PERSQUEUE NOTICE: [PQ: 72075186224037896] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-11-05T14:14:06.216906Z node 4 :PERSQUEUE INFO: [PQ: 72075186224037896] doesn't have tx writes info
2025-11-05T14:14:06.217143Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] server connected, pipe [3:7569244405507278954:2528], now have 1 active actors on pipe
2025-11-05T14:14:06.217173Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] server connected, pipe [3:7569244405507278935:3703], now have 1 active actors on pipe
2025-11-05T14:14:06.218059Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Handle TEvPersQueue::TEvProposeTransaction SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } TxId: 281474976715692 Config { TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc3" Version: 0 LocalDC: true ReadRules: "user" ReadFromTimestampsMs: 0 ConsumerFormatVersions: 0 ConsumerCodecs { } Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } ReadRuleVersions: 0 TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } } }
2025-11-05T14:14:06.218193Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] Propose TxId 281474976715692, WriteId (empty maybe)
2025-11-05T14:14:06.218220Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Try execute txs with state UNKNOWN
2025-11-05T14:14:06.218237Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692, State UNKNOWN
2025-11-05T14:14:06.218257Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] schedule TEvProposeTransactionResult(PREPARED)
2025-11-05T14:14:06.218278Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from UNKNOWN to PREPARING
2025-11-05T14:14:06.218319Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] write key for TxId 281474976715692
2025-11-05T14:14:06.218607Z node 4 :PQ_TX DEBUG: [TxId: 281474976715692] save tx TxId: 281474976715692 State: PREPARED MinStep: 1762352046000 MaxStep: 18446744073709551615 Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc3" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { }
2025-11-05T14:14:06.218681Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:06.217742Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] TEvClientConnected TabletId 72075186224037896, NodeId 4, Generation 1
2025-11-05T14:14:06.221915Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:06.221955Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Try execute txs with state PREPARING
2025-11-05T14:14:06.221972Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692, State PREPARING
2025-11-05T14:14:06.221990Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from PREPARING to PREPARED
2025-11-05T14:14:06.227469Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Handle TEvPersQueue::TEvProposeTransactionAttach TabletId: 72075186224037896 TxId: 281474976715692
2025-11-05T14:14:06.231583Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] server connected, pipe [3:7569244405507278987:3747], now have 1 active actors on pipe
2025-11-05T14:14:06.231840Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Handle TEvTxProcessing::TEvPlanStep Transactions { TxId: 281474976715692 AckTo { RawX1: 0 RawX2: 0 } } Step: 1762352046273 MediatorID: 72057594046382081 TabletID: 72075186224037896
2025-11-05T14:14:06.231881Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Try execute txs with state PREPARED
2025-11-05T14:14:06.231904Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692, State PREPARED
2025-11-05T14:14:06.231927Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from PREPARED to PLANNING
2025-11-05T14:14:06.231955Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] PlanStep 1762352046273, PlanTxId 281474976715692
2025-11-05T14:14:06.232032Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] write key for TxId 281474976715692
2025-11-05T14:14:06.232347Z node 4 :PQ_TX DEBUG: [TxId: 281474976715692] save tx TxId: 281474976715692 State: PLANNED MinStep: 1762352046000 MaxStep: 18446744073709551615 Step: 1762352046273 Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc3" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { }
2025-11-05T14:14:06.232457Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:06.237541Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:06.237586Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Try execute txs with state PLANNING
2025-11-05T14:14:06.237603Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692, State PLANNING
2025-11-05T14:14:06.237621Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from PLANNING to PLANNED
2025-11-05T14:14:06.237641Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxQueue.size 1
2025-11-05T14:14:06.237658Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] New ExecStep 1762352046273, ExecTxId 281474976715692
2025-11-05T14:14:06.238336Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from PLANNED to CALCULATING
2025-11-05T14:14:06.238406Z node 4 :PERSQUEUE DEBUG: [rt3.dc1--test-topic-mirrored-from-dc3:0:Initializer] Start initializing step TInitConfigStep
2025-11-05T14:14:06.238449Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateInit] HandleOnInit TEvPQ::TEvProposePartitionConfig
2025-11-05T14:14:06.238722Z node 4 :PERSQUEUE DEBUG: [rt3.dc1--test-topic-mirrored-from-dc3:0:Initializer] Start initializing step TInitInternalFieldsStep
2025-11-05T14:14:06.238938Z node 4 :PERSQUEUE INFO: [PQ: 72075186224037896, Partition: 0, State: StateInit] bootstrapping 0 [4:7569244402127687432:2440]
2025-11-05T14:14:06.242081Z node 4 :PERSQUEUE DEBUG: [rt3.dc1--test-topic-mirrored-from-dc3:0:Initializer] Initializing completed.
2025-11-05T14:14:06.242120Z node 4 :PERSQUEUE INFO: [PQ: 72075186224037896, Partition: 0, State: StateInit] init complete for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 generation 1 [4:7569244402127687432:2440]
2025-11-05T14:14:06.242153Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateInit] SYNC INIT topic rt3.dc1--test-topic-mirrored-from-dc3 partitition 0 so 0 endOffset 0 Head Offset 0 PartNo 0 PackedSize 0 count 0 nextOffset 0 batches 0
2025-11-05T14:14:06.243330Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Process pending events. Count 1
2025-11-05T14:14:06.243373Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.243452Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.243482Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Still pending for come commits
2025-11-05T14:14:06.243503Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.243525Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.243541Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Still pending for come commits
2025-11-05T14:14:06.243583Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-11-05T14:14:06.243791Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Handle TEvPQ::TEvProposePartitionConfigResult Step 1762352046273, TxId 281474976715692, Partition 0
2025-11-05T14:14:06.243815Z node 4 :PQ_TX DEBUG: [TxId: 281474976715692] Handle TEvProposePartitionConfigResult
2025-11-05T14:14:06.243833Z node 4 :PQ_TX DEBUG: [TxId: 281474976715692] Partition responses 1/1
2025-11-05T14:14:06.243848Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Try execute txs with state CALCULATING
2025-11-05T14:14:06.243864Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692, State CALCULATING
2025-11-05T14:14:06.243880Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692 State CALCULATING FrontTxId 281474976715692
2025-11-05T14:14:06.243894Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Received 1, Expected 1
2025-11-05T14:14:06.243913Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from CALCULATING to CALCULATED
2025-11-05T14:14:06.243936Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] write key for TxId 281474976715692
2025-11-05T14:14:06.244270Z node 4 :PQ_TX DEBUG: [TxId: 281474976715692] save tx TxId: 281474976715692 State: CALCULATED MinStep: 1762352046000 MaxStep: 18446744073709551615 Step: 1762352046273 Predicate: true Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc3" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { Partition { PartitionId: 0 } }
2025-11-05T14:14:06.244449Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:06.251099Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:06.251134Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Try execute txs with state CALCULATED
2025-11-05T14:14:06.251151Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692, State CALCULATED
2025-11-05T14:14:06.251173Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692 State CALCULATED FrontTxId 281474976715692
2025-11-05T14:14:06.251190Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from CALCULATED to WAIT_RS
2025-11-05T14:14:06.251233Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] Send TEvTxProcessing::TEvReadSet to 0 receivers. Wait TEvTxProcessing::TEvReadSet from 0 senders.
2025-11-05T14:14:06.251256Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] HaveParticipantsDecision 1
2025-11-05T14:14:06.251303Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from WAIT_RS to EXECUTING
2025-11-05T14:14:06.251316Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Received 0, Expected 1
2025-11-05T14:14:06.251386Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Handle TEvPQ::TEvTxCommit Step 1762352046273, TxId 281474976715692
2025-11-05T14:14:06.251409Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.251428Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.251560Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.251722Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:06.259170Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:06.259211Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.259230Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.259320Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] Handle TEvPQ::TEvTxCommitDone Step 1762352046273, TxId 281474976715692, Partition 0
2025-11-05T14:14:06.259339Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Try execute txs with state EXECUTING
2025-11-05T14:14:06.259354Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692, State EXECUTING
2025-11-05T14:14:06.259371Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692 State EXECUTING FrontTxId 281474976715692
2025-11-05T14:14:06.259386Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Received 1, Expected 1
2025-11-05T14:14:06.259406Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId: 281474976715692 send TEvPersQueue::TEvProposeTransactionResult(COMPLETE)
2025-11-05T14:14:06.259426Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] complete TxId 281474976715692
2025-11-05T14:14:06.259687Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Apply new config PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc3" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false }
2025-11-05T14:14:06.259787Z node 4 :PERSQUEUE NOTICE: [PQ: 72075186224037896] disable metering: reason# billing is not enabled in BillingMeteringConfig
2025-11-05T14:14:06.259839Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] delete partitions for TxId 281474976715692
2025-11-05T14:14:06.259857Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from EXECUTING to EXECUTED
2025-11-05T14:14:06.259880Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] write key for TxId 281474976715692
2025-11-05T14:14:06.260136Z node 4 :PQ_TX DEBUG: [TxId: 281474976715692] save tx TxId: 281474976715692 State: EXECUTED MinStep: 1762352046000 MaxStep: 18446744073709551615 Step: 1762352046273 Predicate: true Kind: KIND_CONFIG TabletConfig { PartitionConfig { LifetimeSeconds: 86400 LowWatermark: 8388608 SourceIdLifetimeSeconds: 86400 WriteSpeedInBytesPerSecond: 20000000 BurstSize: 20000000 TotalPartitions: 1 SourceIdMaxCounts: 6000000 } PartitionIds: 0 TopicName: "rt3.dc1--test-topic-mirrored-from-dc3" Version: 0 LocalDC: true Codecs { Ids: 0 Ids: 1 Ids: 2 Codecs: "raw" Codecs: "gzip" Codecs: "lzop" } TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3" YcCloudId: "" YcFolderId: "" YdbDatabaseId: "" YdbDatabasePath: "/Root" Partitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } ReadRuleGenerations: 0 AllPartitions { PartitionId: 0 Status: Active CreateVersion: 1 TabletId: 0 } Consumers { Name: "user" ReadFromTimestampsMs: 0 FormatVersion: 0 Codec { } Version: 0 Generation: 0 Important: false } } BootstrapConfig { } SourceActor { RawX1: 7569244349672701814 RawX2: 12884904085 } Partitions { Partition { PartitionId: 0 } }
2025-11-05T14:14:06.260303Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:06.261256Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.261284Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.269032Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:06.269069Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Try execute txs with state EXECUTED
2025-11-05T14:14:06.269085Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692, State EXECUTED
2025-11-05T14:14:06.269102Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692 State EXECUTED FrontTxId 281474976715692
2025-11-05T14:14:06.269119Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TPersQueue::SendEvReadSetAckToSenders
2025-11-05T14:14:06.269136Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from EXECUTED to WAIT_RS_ACKS
2025-11-05T14:14:06.269161Z node 4 :PQ_TX DEBUG: [TxId: 281474976715692] PredicateAcks: 0/0
2025-11-05T14:14:06.269170Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] HaveAllRecipientsReceive 1, AllSupportivePartitionsHaveBeenDeleted 1
2025-11-05T14:14:06.269182Z node 4 :PQ_TX DEBUG: [TxId: 281474976715692] PredicateAcks: 0/0
2025-11-05T14:14:06.269196Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] add an TxId 281474976715692 to the list for deletion
2025-11-05T14:14:06.269213Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] TxId 281474976715692 moved from WAIT_RS_ACKS to DELETING
2025-11-05T14:14:06.269237Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] delete key for TxId 281474976715692
2025-11-05T14:14:06.269286Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Send TEvKeyValue::TEvRequest (WRITE_TX_COOKIE)
2025-11-05T14:14:06.272626Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Handle TEvKeyValue::TEvResponse (WRITE_TX_COOKIE)
2025-11-05T14:14:06.272659Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] Try execute txs with state DELETING
2025-11-05T14:14:06.272689Z node 4 :PQ_TX DEBUG: [PQ: 72075186224037896] TxId 281474976715692, State DELETING
2025-11-05T14:14:06.272710Z node 4 :PQ_TX INFO: [PQ: 72075186224037896] delete TxId 281474976715692
2025-11-05T14:14:06.296647Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.296704Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.343722Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.343773Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.359817Z node 3 :KQP_EXECUTER ERROR: TxId: 281474976715693. Ctx: { TraceId: 01k9a5w84c16g9jrr58zn8hhr8, Database: , DatabaseId: /Root, SessionId: ydb://session/3?node_id=3&id=N2M0NGQ3MDAtYzY2ZTIwOC03MTkyODY2Zi1mNmM5MTZmNQ==, CurrentExecutionId: , CustomerSuppliedId: , PoolId: }. Database not set, use /Root
2025-11-05T14:14:06.361598Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.361635Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
===Query complete
TClient::Ls request: /Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3
TClient::Ls response: Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic-mirrored-from-dc3"
    PathId: 11
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976715692
    CreateStep: 1762352046273
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
    BalancerTabletID: 72075186224037897
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic-mirrored-from-dc3"
    PathId: 11
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBy...
(TRUNCATED)
GetTopicVersionFromPath:  record Status: 1
StatusCode: SUCCESS
SchemeStatus: 0
PathDescription {
  Self {
    Name: "rt3.dc1--test-topic-mirrored-from-dc3"
    PathId: 11
    SchemeshardId: 72057594046644480
    PathType: EPathTypePersQueueGroup
    CreateFinished: true
    CreateTxId: 281474976715692
    CreateStep: 1762352046273
    ParentPathId: 2
    PathState: EPathStateNoChanges
    Owner: "root@builtin"
    ACL: ""
    EffectiveACL: ""
    PathVersion: 2
    PathSubType: EPathSubTypeEmpty
    Version {
      GeneralVersion: 2
      ACLVersion: 0
      EffectiveACLVersion: 0
      UserAttrsVersion: 1
      ChildrenVersion: 0
      PQVersion: 1
    }
    ChildrenExist: false
    BalancerTabletID: 72075186224037897
  }
  PersQueueGroup {
    Name: "rt3.dc1--test-topic-mirrored-from-dc3"
    PathId: 11
    TotalGroupCount: 1
    PartitionPerTablet: 5
    PQTabletConfig {
      PartitionConfig {
        LifetimeSeconds: 86400
        LowWatermark: 8388608
        SourceIdLifetimeSeconds: 86400
        WriteSpeedInBytesPerSecond: 20000000
        BurstSize: 20000000
        SourceIdMaxCounts: 6000000
      }
      LocalDC: true
      ReadRules: "user"
      ReadFromTimestampsMs: 0
      ConsumerFormatVersions: 0
      ConsumerCodecs {
      }
      Codecs {
        Ids: 0
        Ids: 1
        Ids: 2
        Codecs: "raw"
        Codecs: "gzip"
        Codecs: "lzop"
      }
      ReadRuleVersions: 0
      YdbDatabasePath: "/Root"
    }
    Partitions {
      PartitionId: 0
      TabletId: 72075186224037896
      Status: Active
    }
    AlterVersion: 1
    BalancerTabletID: 72075186224037897
    NextPartitionId: 1
  }
  DomainDescription {
    SchemeShardId_Depricated: 72057594046644480
    PathId_Depricated: 1
    ProcessingParams {
      Version: 1
      PlanResolution: 7
      Coordinators: 72057594046316545
      TimeCastBucketsPerMediator: 2
      Mediators: 72057594046382081
    }
    DomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    PathsInside: 10
    PathsLimit: 10000
    ShardsInside: 10
    ShardsLimit: 200000
    ResourcesDomainKey {
      SchemeShard: 72057594046644480
      PathId: 1
    }
    DiskSpaceUsage {
      Tables {
        TotalSize: 0
        DataSize: 0
        IndexSize: 0
      }
      Topics {
        ReserveSize: 0
        AccountSize: 0
        DataSize: 0
        UsedReserveSize: 0
      }
    }
    PQPartitionsInside: 3
    PQPartitionsLimit: 1000000
  }
}
Path: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3"
 name rt3.dc1--test-topic-mirrored-from-dc3 version1
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc3"
  }
}
2025-11-05T14:14:06.380584Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 128
ErrorReason: "the following topics are not created: rt3.dc1--test-topic-mirrored-from-dc3, Marker# PQ95"
ErrorCode: UNKNOWN_TOPIC
2025-11-05T14:14:06.397303Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.397341Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.444109Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.444150Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.461983Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.462023Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.497277Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.497317Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.546103Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.546145Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.562346Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.562390Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.598534Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.598579Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.646495Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.646536Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.662807Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.662848Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.699468Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.699537Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.746624Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.746654Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.763082Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.763130Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.799410Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.799448Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.847043Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.847088Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.863655Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.863702Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
CallPersQueueGRPC request to localhost:20226
MetaRequest {
  CmdGetTopicMetadata {
    Topic: "rt3.dc1--test-topic-mirrored-from-dc3"
  }
}
2025-11-05T14:14:06.887190Z node 3 :PERSQUEUE INFO: proxy answer <none>
CallPersQueueGRPC response:
Status: 1
ErrorCode: OK
MetaResponse {
  CmdGetTopicMetadataResult {
    TopicInfo {
      Topic: "rt3.dc1--test-topic-mirrored-from-dc3"
      NumPartitions: 1
      Config {
        PartitionConfig {
          LifetimeSeconds: 86400
          LowWatermark: 8388608
          SourceIdLifetimeSeconds: 86400
          WriteSpeedInBytesPerSecond: 20000000
          BurstSize: 20000000
          SourceIdMaxCounts: 6000000
        }
        Version: 1
        LocalDC: true
        Codecs {
          Ids: 0
          Ids: 1
          Ids: 2
          Codecs: "raw"
          Codecs: "gzip"
          Codecs: "lzop"
        }
        TopicPath: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3"
        YdbDatabasePath: "/Root"
        Consumers {
          Name: "user"
          ReadFromTimestampsMs: 0
          FormatVersion: 0
          Codec {
          }
          Version: 0
          Important: false
        }
      }
      ErrorCode: OK
    }
  }
}
=== Topic created, have version: 1
2025-11-05T14:14:06.899739Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.899781Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
>>> Session was created
2025-11-05T14:14:06.947363Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.947408Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
>>> Ready to answer: ok
2025-11-05T14:14:06.954989Z :INFO: [/Root] [] [545e2055-e44d2782-d5d23efa-dd6b2e47] Open read subsessions to databases: { name: dc1, endpoint: localhost:20226, path: /Root }
2025-11-05T14:14:06.963733Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.963771Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.964067Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] Starting read session
2025-11-05T14:14:06.964119Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] Starting single session
2025-11-05T14:14:06.966851Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In Reconnect, ReadSizeBudget = 8388608, ReadSizeServerDelta = 0
2025-11-05T14:14:06.967256Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] New values: ReadSizeBudget = 8388608, ReadSizeServerDelta = 0
2025-11-05T14:14:06.967299Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Reconnecting session to cluster  in 0.000000s
2025-11-05T14:14:06.967354Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: try to update token
2025-11-05T14:14:06.974004Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session: Do CDS request
2025-11-05T14:14:06.974102Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Start write session. Will connect to endpoint: localhost:20226
2025-11-05T14:14:06.977282Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Successfully connected. Initializing session
2025-11-05T14:14:06.978601Z node 3 :PQ_READ_PROXY DEBUG: new grpc connection
2025-11-05T14:14:06.978629Z node 3 :PQ_READ_PROXY DEBUG: new session created cookie 1
2025-11-05T14:14:06.981947Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer  session  grpc read done: success# 1, data# { init_request { topics_read_settings { path: "test-topic" } topics_read_settings { path: "test-topic-mirrored-from-dc2" } topics_read_settings { path: "test-topic-mirrored-from-dc3" } consumer: "shared/user" } }
2025-11-05T14:14:06.982345Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 read init: from# ipv6:[::1]:45292, request# { init_request { topics_read_settings { path: "test-topic" } topics_read_settings { path: "test-topic-mirrored-from-dc2" } topics_read_settings { path: "test-topic-mirrored-from-dc3" } consumer: "shared/user" } }
2025-11-05T14:14:06.982799Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 auth for : user
2025-11-05T14:14:06.983764Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Handle describe topics response
2025-11-05T14:14:06.984001Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 auth is DEAD
2025-11-05T14:14:06.984100Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 auth ok: topics# 3, initDone# 0
2025-11-05T14:14:06.985125Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 register session: topic# rt3.dc1--test-topic-mirrored-from-dc3
2025-11-05T14:14:06.985161Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 register session: topic# rt3.dc1--test-topic
2025-11-05T14:14:06.985175Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 register session: topic# rt3.dc1--test-topic-mirrored-from-dc2
2025-11-05T14:14:06.986754Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Server session id: shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:06.986796Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 8388608, ReadSizeServerDelta = 0
2025-11-05T14:14:06.987081Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-11-05T14:14:06.985657Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7569244405507279149:2559] connected; active server actors: 1
2025-11-05T14:14:06.985694Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer "user" register session for pipe [3:7569244405507279149:2559] session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:06.985734Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user register readable partition 0
2025-11-05T14:14:06.985784Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user family created family=1 (Status=Free, Partitions=[0])
2025-11-05T14:14:06.985820Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user register reading session ReadingSession "shared/user_3_1_5731498468557581848_v1" (Sender=[3:7569244405507279146:2559], Pipe=[3:7569244405507279149:2559], Partitions=[], ActiveFamilyCount=0)
2025-11-05T14:14:06.985843Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user rebalancing was scheduled
2025-11-05T14:14:06.985888Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user balancing. Sessions=1, Families=1, UnradableFamilies=1 [1 (0), ], RequireBalancing=0 []
2025-11-05T14:14:06.985919Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user balancing family=1 (Status=Free, Partitions=[0]) for ReadingSession "shared/user_3_1_5731498468557581848_v1" (Sender=[3:7569244405507279146:2559], Pipe=[3:7569244405507279149:2559], Partitions=[], ActiveFamilyCount=0)
2025-11-05T14:14:06.985995Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user family 1 status Active partitions [0] session "shared/user_3_1_5731498468557581848_v1" sender [3:7569244405507279146:2559] lock partition 0 for ReadingSession "shared/user_3_1_5731498468557581848_v1" (Sender=[3:7569244405507279146:2559], Pipe=[3:7569244405507279149:2559], Partitions=[], ActiveFamilyCount=1) generation 1 step 1
2025-11-05T14:14:06.986026Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user start rebalancing. familyCount=1, sessionCount=1, desiredFamilyCount=1, allowPlusOne=0
2025-11-05T14:14:06.986043Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] consumer user balancing duration: 0.000140s
2025-11-05T14:14:06.986114Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] pipe [3:7569244405507279150:2559] connected; active server actors: 1
2025-11-05T14:14:06.986133Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: [72075186224037893][rt3.dc1--test-topic] consumer "user" register session for pipe [3:7569244405507279150:2559] session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:06.986151Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] consumer user register readable partition 0
2025-11-05T14:14:06.986168Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] consumer user family created family=1 (Status=Free, Partitions=[0])
2025-11-05T14:14:06.986184Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] consumer user register reading session ReadingSession "shared/user_3_1_5731498468557581848_v1" (Sender=[3:7569244405507279146:2559], Pipe=[3:7569244405507279150:2559], Partitions=[], ActiveFamilyCount=0)
2025-11-05T14:14:06.986194Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] consumer user rebalancing was scheduled
2025-11-05T14:14:06.986214Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] consumer user balancing. Sessions=1, Families=1, UnradableFamilies=1 [1 (0), ], RequireBalancing=0 []
2025-11-05T14:14:06.986237Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] consumer user balancing family=1 (Status=Free, Partitions=[0]) for ReadingSession "shared/user_3_1_5731498468557581848_v1" (Sender=[3:7569244405507279146:2559], Pipe=[3:7569244405507279150:2559], Partitions=[], ActiveFamilyCount=0)
2025-11-05T14:14:06.986264Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] consumer user family 1 status Active partitions [0] session "shared/user_3_1_5731498468557581848_v1" sender [3:7569244405507279146:2559] lock partition 0 for ReadingSession "shared/user_3_1_5731498468557581848_v1" (Sender=[3:7569244405507279146:2559], Pipe=[3:7569244405507279150:2559], Partitions=[], ActiveFamilyCount=1) generation 1 step 1
2025-11-05T14:14:06.986287Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] consumer user start rebalancing. familyCount=1, sessionCount=1, desiredFamilyCount=1, allowPlusOne=0
2025-11-05T14:14:06.986298Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037893][rt3.dc1--test-topic] consumer user balancing duration: 0.000074s
2025-11-05T14:14:06.986321Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7569244405507279151:2559] connected; active server actors: 1
2025-11-05T14:14:06.986337Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer "user" register session for pipe [3:7569244405507279151:2559] session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:06.986357Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user register readable partition 0
2025-11-05T14:14:06.986383Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user family created family=1 (Status=Free, Partitions=[0])
2025-11-05T14:14:06.986404Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user register reading session ReadingSession "shared/user_3_1_5731498468557581848_v1" (Sender=[3:7569244405507279146:2559], Pipe=[3:7569244405507279151:2559], Partitions=[], ActiveFamilyCount=0)
2025-11-05T14:14:06.986420Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user rebalancing was scheduled
2025-11-05T14:14:06.986462Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user balancing. Sessions=1, Families=1, UnradableFamilies=1 [1 (0), ], RequireBalancing=0 []
2025-11-05T14:14:06.986496Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user balancing family=1 (Status=Free, Partitions=[0]) for ReadingSession "shared/user_3_1_5731498468557581848_v1" (Sender=[3:7569244405507279146:2559], Pipe=[3:7569244405507279151:2559], Partitions=[], ActiveFamilyCount=0)
2025-11-05T14:14:06.986545Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user family 1 status Active partitions [0] session "shared/user_3_1_5731498468557581848_v1" sender [3:7569244405507279146:2559] lock partition 0 for ReadingSession "shared/user_3_1_5731498468557581848_v1" (Sender=[3:7569244405507279146:2559], Pipe=[3:7569244405507279151:2559], Partitions=[], ActiveFamilyCount=1) generation 1 step 1
2025-11-05T14:14:06.986567Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user start rebalancing. familyCount=1, sessionCount=1, desiredFamilyCount=1, allowPlusOne=0
2025-11-05T14:14:06.986587Z node 3 :PERSQUEUE_READ_BALANCER DEBUG: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] consumer user balancing duration: 0.000107s
2025-11-05T14:14:06.987324Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 assign: record# { Partition: 0 TabletId: 72075186224037896 Topic: "rt3.dc1--test-topic-mirrored-from-dc3" Generation: 1 Step: 1 Session: "shared/user_3_1_5731498468557581848_v1" ClientId: "user" PipeClient { RawX1: 7569244405507279149 RawX2: 4503612512274943 } Path: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc3" }
2025-11-05T14:14:06.987452Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 INITING TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1)
2025-11-05T14:14:06.987911Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 assign: record# { Partition: 0 TabletId: 72075186224037892 Topic: "rt3.dc1--test-topic" Generation: 1 Step: 1 Session: "shared/user_3_1_5731498468557581848_v1" ClientId: "user" PipeClient { RawX1: 7569244405507279150 RawX2: 4503612512274943 } Path: "/Root/PQ/rt3.dc1--test-topic" }
2025-11-05T14:14:06.988502Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 assign: record# { Partition: 0 TabletId: 72075186224037894 Topic: "rt3.dc1--test-topic-mirrored-from-dc2" Generation: 1 Step: 1 Session: "shared/user_3_1_5731498468557581848_v1" ClientId: "user" PipeClient { RawX1: 7569244405507279151 RawX2: 4503612512274943 } Path: "/Root/PQ/rt3.dc1--test-topic-mirrored-from-dc2" }
2025-11-05T14:14:06.988579Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { read_request { bytes_size: 8388608 } }
2025-11-05T14:14:06.988655Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got read request: guid# 6aa6470a-9399948f-7768b09-c9cd0f0d
2025-11-05T14:14:06.989112Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 INITING TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2)
2025-11-05T14:14:06.989182Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 INITING TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3)
2025-11-05T14:14:06.989605Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] server connected, pipe [3:7569244405507279156:2562], now have 1 active actors on pipe
2025-11-05T14:14:06.989771Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) pipe restart attempt 0 pipe creation result: OK TabletId: 72075186224037896 Generation: 1, pipe: [3:7569244405507279156:2562]
2025-11-05T14:14:06.989897Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:06.989926Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:06.989948Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Created session shared/user_3_1_5731498468557581848_v1 on pipe: [3:7569244405507279156:2562]
2025-11-05T14:14:06.990007Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.990042Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.990066Z node 4 :PQ_READ_PROXY DEBUG: Direct read cache: registered server session: shared/user_3_1_5731498468557581848_v1:1 with generation 1
2025-11-05T14:14:06.990109Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user session is set to 0 (startOffset 0) session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:06.990124Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.990252Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:06.992323Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] server connected, pipe [3:7569244405507279162:2565], now have 1 active actors on pipe
2025-11-05T14:14:06.992357Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] server connected, pipe [3:7569244405507279163:2566], now have 1 active actors on pipe
2025-11-05T14:14:06.992476Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) pipe restart attempt 0 pipe creation result: OK TabletId: 72075186224037892 Generation: 1, pipe: [3:7569244405507279162:2565]
2025-11-05T14:14:06.992549Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) pipe restart attempt 0 pipe creation result: OK TabletId: 72075186224037894 Generation: 1, pipe: [3:7569244405507279163:2566]
2025-11-05T14:14:06.992736Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:06.992766Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:06.992795Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Created session shared/user_3_1_5731498468557581848_v1 on pipe: [3:7569244405507279162:2565]
2025-11-05T14:14:06.992850Z node 4 :PQ_READ_PROXY DEBUG: Direct read cache: registered server session: shared/user_3_1_5731498468557581848_v1:2 with generation 1
2025-11-05T14:14:06.992904Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.992931Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.992989Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 user user session is set to 0 (startOffset 0) session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:06.993005Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.993161Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:06.993191Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:06.993213Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Created session shared/user_3_1_5731498468557581848_v1 on pipe: [3:7569244405507279163:2566]
2025-11-05T14:14:06.993248Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:06.993279Z node 4 :PQ_READ_PROXY DEBUG: Direct read cache: registered server session: shared/user_3_1_5731498468557581848_v1:3 with generation 1
2025-11-05T14:14:06.993315Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:06.993340Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.993385Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user session is set to 0 (startOffset 0) session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:06.993399Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:06.993483Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:06.995129Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: send init request: init_request { topic: "test-topic" message_group_id: "src_id" }
2025-11-05T14:14:06.995591Z node 3 :PQ_WRITE_PROXY DEBUG: new grpc connection
2025-11-05T14:14:06.995617Z node 3 :PQ_WRITE_PROXY DEBUG: new session created cookie 2
2025-11-05T14:14:06.997691Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 2 sessionId:  grpc read done: success: 1 data: init_request { topic: "test-topic" message_group_id: "src_id" }
2025-11-05T14:14:06.997847Z node 3 :PQ_WRITE_PROXY INFO: session request cookie: 2 topic: "test-topic" message_group_id: "src_id" from ipv6:[::1]:45292
2025-11-05T14:14:06.997861Z node 3 :PQ_WRITE_PROXY INFO: write session:  cookie=2 sessionId= userAgent="pqv1 server" ip=ipv6:[::1]:45292 proto=v1  topic=test-topic durationSec=0
2025-11-05T14:14:06.997872Z node 3 :PQ_WRITE_PROXY INFO: init check schema
2025-11-05T14:14:07.000900Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:06.998993Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 2 sessionId:  describe result for acl check
2025-11-05T14:14:06.999081Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper SelectQuery: --!syntax_v1
DECLARE $Hash AS Uint32; DECLARE $Topic AS Utf8; DECLARE $SourceId AS Utf8;
SELECT Partition, CreateTime, AccessTime, SeqNo FROM `/Root/PQ/SourceIdMeta2` WHERE Hash == $Hash AND Topic == $Topic AND SourceId == $SourceId;
2025-11-05T14:14:06.999088Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper UpdateQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;DECLARE $SeqNo AS Uint64;
UPSERT INTO `/Root/PQ/SourceIdMeta2` (Hash, Topic, SourceId, CreateTime, AccessTime, Partition, SeqNo) VALUES ($Hash, $Topic, $SourceId, $CreateTime, $AccessTime, $Partition, $SeqNo);
2025-11-05T14:14:06.999093Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper UpdateAccessTimeQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;
UPDATE `/Root/PQ/SourceIdMeta2` SET AccessTime = $AccessTime WHERE Hash = $Hash AND Topic = $Topic AND SourceId = $SourceId AND Partition = $Partition;
2025-11-05T14:14:06.999105Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244405507279169:2567] (SourceId=src_id, PreferedPartition=(NULL)) StartKqpSession
2025-11-05T14:14:07.000805Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244405507279169:2567] (SourceId=src_id, PreferedPartition=(NULL)) Select from the table
2025-11-05T14:14:07.002364Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:07.002364Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:07.002399Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.002403Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.002419Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.002420Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.002475Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 18446744073709551615
2025-11-05T14:14:07.002500Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 0 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:07.002518Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.002524Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 18446744073709551615
2025-11-05T14:14:07.002532Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.002549Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 18446744073709551615
2025-11-05T14:14:07.004298Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Confirm partition stream create. Partition stream id: 1. Cluster: "-". Topic: "test-topic". Partition: 0. Read offset: (NULL)
2025-11-05T14:14:07.004600Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Confirm partition stream create. Partition stream id: 2. Cluster: "-". Topic: "test-topic-mirrored-from-dc2". Partition: 0. Read offset: (NULL)
2025-11-05T14:14:07.004750Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Confirm partition stream create. Partition stream id: 3. Cluster: "-". Topic: "test-topic-mirrored-from-dc3". Partition: 0. Read offset: (NULL)
2025-11-05T14:14:07.003125Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) initDone 0 event { CmdGetClientOffsetResult { Offset: 0 EndOffset: 0 WriteTimestampMS: 1762352043937 CreateTimestampMS: 1762352043937 SizeLag: 0 WriteTimestampEstimateMS: 1762352046963 ClientHasAnyCommits: false } Cookie: 18446744073709551615 }
2025-11-05T14:14:07.003170Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 INIT DONE TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) EndOffset 0 readOffset 0 committedOffset 0
2025-11-05T14:14:07.003267Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) initDone 0 event { CmdGetClientOffsetResult { Offset: 0 EndOffset: 0 WriteTimestampMS: 1762352045489 CreateTimestampMS: 1762352045489 SizeLag: 0 WriteTimestampEstimateMS: 1762352046899 ClientHasAnyCommits: false } Cookie: 18446744073709551615 }
2025-11-05T14:14:07.003289Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 INIT DONE TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) EndOffset 0 readOffset 0 committedOffset 0
2025-11-05T14:14:07.003367Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 sending to client partition status
2025-11-05T14:14:07.003649Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 sending to client partition status
2025-11-05T14:14:07.004081Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) initDone 0 event { CmdGetClientOffsetResult { Offset: 0 EndOffset: 0 WriteTimestampMS: 1762352046242 CreateTimestampMS: 1762352046242 SizeLag: 0 WriteTimestampEstimateMS: 1762352046947 ClientHasAnyCommits: false } Cookie: 18446744073709551615 }
2025-11-05T14:14:07.004109Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 INIT DONE TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) EndOffset 0 readOffset 0 committedOffset 0
2025-11-05T14:14:07.004149Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 sending to client partition status
2025-11-05T14:14:07.005276Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { start_partition_session_response { partition_session_id: 2 } }
2025-11-05T14:14:07.005421Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got StartRead from client: partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), readOffset# 0, commitOffset# (empty maybe)
2025-11-05T14:14:07.005493Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { start_partition_session_response { partition_session_id: 3 } }
2025-11-05T14:14:07.005527Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Start reading TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) EndOffset 0 readOffset 0 committedOffset 0 clientCommitOffset (empty maybe) clientReadOffset 0
2025-11-05T14:14:07.005563Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait data in partition inited, cookie 1 from offset0
2025-11-05T14:14:07.005566Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got StartRead from client: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), readOffset# 0, commitOffset# (empty maybe)
2025-11-05T14:14:07.005598Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { start_partition_session_response { partition_session_id: 1 } }
2025-11-05T14:14:07.005637Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Start reading TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) EndOffset 0 readOffset 0 committedOffset 0 clientCommitOffset (empty maybe) clientReadOffset 0
2025-11-05T14:14:07.005652Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait data in partition inited, cookie 1 from offset0
2025-11-05T14:14:07.005720Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got StartRead from client: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), readOffset# 0, commitOffset# (empty maybe)
2025-11-05T14:14:07.005757Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Start reading TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) EndOffset 0 readOffset 0 committedOffset 0 clientCommitOffset (empty maybe) clientReadOffset 0
2025-11-05T14:14:07.005932Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait data in partition inited, cookie 1 from offset0
2025-11-05T14:14:07.014812Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244405507279169:2567] (SourceId=src_id, PreferedPartition=(NULL)) RequestPQRB
2025-11-05T14:14:07.015224Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] pipe [3:7569244409802246482:2567] connected; active server actors: 1
2025-11-05T14:14:07.015290Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244405507279169:2567] (SourceId=src_id, PreferedPartition=(NULL)) Received partition 0 from PQRB for SourceId=src_id
2025-11-05T14:14:07.015309Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244405507279169:2567] (SourceId=src_id, PreferedPartition=(NULL)) Update the table
2025-11-05T14:14:07.015606Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] pipe [3:7569244409802246482:2567] disconnected; active server actors: 1
2025-11-05T14:14:07.015625Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] pipe [3:7569244409802246482:2567] disconnected no session
2025-11-05T14:14:07.026632Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244405507279169:2567] (SourceId=src_id, PreferedPartition=(NULL)) HandleUpdate PartitionPersisted=0 Status=SUCCESS
2025-11-05T14:14:07.026662Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244405507279169:2567] (SourceId=src_id, PreferedPartition=(NULL)) ReplyResult: Partition=0, SeqNo=(NULL)
2025-11-05T14:14:07.026674Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244405507279169:2567] (SourceId=src_id, PreferedPartition=(NULL)) Start idle
2025-11-05T14:14:07.026696Z node 3 :PQ_WRITE_PROXY DEBUG: ProceedPartition. session cookie: 2 sessionId:  partition: 0 expectedGeneration: (NULL)
2025-11-05T14:14:07.027737Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] server connected, pipe [3:7569244409802246495:2567], now have 1 active actors on pipe
2025-11-05T14:14:07.027998Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.028037Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.028125Z node 4 :PERSQUEUE INFO: new Cookie src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0 generated for partition 0 topic 'rt3.dc1--test-topic' owner src_id
2025-11-05T14:14:07.027847Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) TEvClientConnected Status OK, TabletId: 72075186224037892, NodeId 4, Generation: 1
2025-11-05T14:14:07.028169Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.028198Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.028237Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.028279Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::ReplyOwnerOk. Partition: 0
2025-11-05T14:14:07.028362Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:07.028787Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.028813Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.028921Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:07.029163Z node 3 :PQ_WRITE_PROXY INFO: session inited cookie: 2 partition: 0 MaxSeqNo: 0 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0
2025-11-05T14:14:07.029986Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1762352047029 BytesWritten: 0 MessagesWritten: 0 BytesWrittenCompressed: 0 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-11-05T14:14:07.030111Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session established. Init response: session_id: "src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0" topic: "test-topic" cluster: "dc1" supported_codecs: CODEC_RAW supported_codecs: CODEC_GZIP supported_codecs: CODEC_LZOP
2025-11-05T14:14:07.030430Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write 1 messages with Id from 1 to 1
2025-11-05T14:14:07.030537Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: try to update token
2025-11-05T14:14:07.030579Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Send 1 message(s) (0 left), first sequence number is 1
2025-11-05T14:14:07.030796Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write 1 messages with Id from 2 to 2
2025-11-05T14:14:07.030873Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: try to update token
2025-11-05T14:14:07.030911Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Send 1 message(s) (0 left), first sequence number is 2
2025-11-05T14:14:07.031062Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write 1 messages with Id from 3 to 3
2025-11-05T14:14:07.031104Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: try to update token
2025-11-05T14:14:07.031121Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Send 1 message(s) (0 left), first sequence number is 3
2025-11-05T14:14:07.031223Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write 1 messages with Id from 4 to 4
2025-11-05T14:14:07.031258Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: try to update token
2025-11-05T14:14:07.031274Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Send 1 message(s) (0 left), first sequence number is 4
2025-11-05T14:14:07.031415Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write 1 messages with Id from 5 to 5
2025-11-05T14:14:07.031450Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: try to update token
2025-11-05T14:14:07.031470Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Send 1 message(s) (0 left), first sequence number is 5
2025-11-05T14:14:07.031570Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: close. Timeout = 18446744073709551 ms
2025-11-05T14:14:07.031896Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 2 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.032122Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 2 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.032284Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.032368Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 2 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.032577Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.032611Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 2 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.032721Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.032757Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.032810Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.032834Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.032859Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-11-05T14:14:07.032911Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 1  requestId: <none> cookie: 2
2025-11-05T14:14:07.033039Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.033059Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.033120Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 2  requestId: <none> cookie: 3
2025-11-05T14:14:07.033216Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.033240Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.033322Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.033344Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.032789Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.032829Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 2 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.032887Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.033040Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.033090Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.033172Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.033250Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.033709Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 1 partNo : 0 messageNo: 4 size 91 offset: -1
2025-11-05T14:14:07.033760Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.033771Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.033788Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 2 partNo : 0 messageNo: 5 size 202 offset: -1
2025-11-05T14:14:07.033801Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 3  requestId: <none> cookie: 4
2025-11-05T14:14:07.033824Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.033829Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.033838Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.033858Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.033896Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.033906Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.033922Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 3 partNo : 0 messageNo: 7 size 312 offset: -1
2025-11-05T14:14:07.033976Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 1 partNo 0
2025-11-05T14:14:07.034205Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.034462Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.034494Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.034513Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 4 partNo : 0 messageNo: 8 size 422 offset: -1
2025-11-05T14:14:07.049605Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.049649Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.087061Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 1 partNo 0 FormedBlobsCount 0 NewHead: Offset 0 PartNo 0 PackedSize 165 count 1 nextOffset 1 batches 1
2025-11-05T14:14:07.087135Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.087643Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Add new write blob: topic 'rt3.dc1--test-topic' partition 0 compactOffset 0,1 HeadOffset 0 endOffset 0 curOffset 1 d0000000000_00000000000000000000_00000_0000000001_00000| size 153 WTime 1762352047087
2025-11-05T14:14:07.087808Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:07.087845Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.087895Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.087906Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough blob. Partition 0 offset 0 partNo 0 count 1 size 153
2025-11-05T14:14:07.087934Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.087970Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.088004Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 6  requestId: <none> cookie: 5
2025-11-05T14:14:07.088335Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.089163Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.089199Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.089244Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message topic: rt3.dc1--test-topic partition: 0 SourceId: '\0src_id' SeqNo: 5 partNo : 0 messageNo: 9 size 532 offset: -1
2025-11-05T14:14:07.089333Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.090826Z node 4 :PERSQUEUE DEBUG: Caching head blob in L1. Partition 0 offset 0 count 1 size 153 actorID [4:7569244389242785200:2392]
2025-11-05T14:14:07.090914Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 98 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:07.090960Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.091015Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 1, partNo: 0, Offset: 0 is stored on disk
2025-11-05T14:14:07.091130Z node 4 :PERSQUEUE DEBUG: PQ Cache (L2). Adding blob. Tablet '72075186224037892' partition 0 offset 0 partno 0 count 1 parts 0 size 153
2025-11-05T14:14:07.091186Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-11-05T14:14:07.091223Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 user user send read request for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 1 rrg 0
2025-11-05T14:14:07.091260Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.091305Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.091487Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait for data done:  readOffset 0 EndOffset 0 newEndOffset 1 commitOffset 0 clientCommitOffset 0 cookie 1 readingFinished 0 firstRead 1
2025-11-05T14:14:07.091529Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) ready for read with readOffset 0 endOffset 1
2025-11-05T14:14:07.091617Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), readOffset# 0, endOffset# 1, WTime# 1762352043937, sizeLag# 153
2025-11-05T14:14:07.091631Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1TEvPartitionReady. Aval parts: 1
2025-11-05T14:14:07.091668Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.091692Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 performing read request: guid# 5559b60a-c2b7303a-cb18b9b4-4a253f92, from# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), count# 1, size# 183, partitionsAsked# 1, maxTimeLag# 0ms
2025-11-05T14:14:07.091781Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 READ FROM TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2)maxCount 1 maxSize 183 maxTimeLagMs 0 readTimestampMs 0 readOffset 0 EndOffset 1 ClientCommitOffset 0 committedOffset 0 Guid 5559b60a-c2b7303a-cb18b9b4-4a253f92
2025-11-05T14:14:07.091408Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 2 partNo 0
2025-11-05T14:14:07.091455Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 4  requestId: <none> cookie: 1
2025-11-05T14:14:07.092159Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.092198Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.092439Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session got write response: sequence_numbers: 1 offsets: 0 already_written: false write_statistics { persist_duration_ms: 3 queued_in_partition_duration_ms: 53 }
2025-11-05T14:14:07.092400Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 2 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 276 count 1 nextOffset 2 batches 1
2025-11-05T14:14:07.092472Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 3 partNo 0
2025-11-05T14:14:07.092489Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: acknoledged message 1
2025-11-05T14:14:07.092511Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 3 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 630 count 2 nextOffset 3 batches 1
2025-11-05T14:14:07.092554Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 4 partNo 0
2025-11-05T14:14:07.092589Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 4 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1094 count 3 nextOffset 4 batches 1
2025-11-05T14:14:07.092625Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob processing sourceId '\0src_id' seqNo 5 partNo 0
2025-11-05T14:14:07.092655Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic' partition 0 part blob complete sourceId '\0src_id' seqNo 5 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1668 count 4 nextOffset 5 batches 1
2025-11-05T14:14:07.092696Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.093219Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Add new write blob: topic 'rt3.dc1--test-topic' partition 0 compactOffset 1,4 HeadOffset 0 endOffset 1 curOffset 5 d0000000000_00000000000000000001_00000_0000000004_00000| size 1600 WTime 1762352047092
2025-11-05T14:14:07.093376Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:07.093500Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough blob. Partition 0 offset 1 partNo 0 count 4 size 1600
2025-11-05T14:14:07.093535Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] read cookie 0 Topic 'rt3.dc1--test-topic' partition 0 user user offset 0 count 1 size 1024000 endOffset 1 max time lag 0ms effective offset 0
2025-11-05T14:14:07.093569Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] read cookie 0 added 0 blobs, size 0 count 0 last offset 0, current partition end offset: 1
2025-11-05T14:14:07.093607Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Reading cookie 0. All data is from uncompacted head.
2025-11-05T14:14:07.093627Z node 4 :PERSQUEUE DEBUG: FormAnswer for 0 blobs
2025-11-05T14:14:07.093689Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] read cookie 1 Topic 'rt3.dc1--test-topic' partition 0 user user offset 0 count 1 size 183 endOffset 1 max time lag 0ms effective offset 0
2025-11-05T14:14:07.093715Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] read cookie 1 added 0 blobs, size 0 count 0 last offset 0, current partition end offset: 1
2025-11-05T14:14:07.093737Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Reading cookie 1. All data is from uncompacted head.
2025-11-05T14:14:07.093746Z node 4 :PERSQUEUE DEBUG: FormAnswer for 0 blobs
2025-11-05T14:14:07.093804Z node 4 :PERSQUEUE DEBUG: Topic 'rt3.dc1--test-topic' partition 0 user user readTimeStamp done, result 1762352047033 queuesize 0 startOffset 0
2025-11-05T14:14:07.093855Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:07.094297Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) initDone 1 event { CmdReadResult { MaxOffset: 1 Result { Offset: 0 Data: "... 91 bytes ..." SourceId: "\000src_id" SeqNo: 1 WriteTimestampMS: 1762352047033 CreateTimestampMS: 1762352047030 UncompressedSize: 20 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 0 SizeLag: 20 RealReadOffset: 0 WaitQuotaTimeMs: 0 EndOffset: 1 StartOffset: 0 } Cookie: 0 }
2025-11-05T14:14:07.094412Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait data in partition inited, cookie 2 from offset1
2025-11-05T14:14:07.094451Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 after read state TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) EndOffset 1 ReadOffset 1 ReadGuid 5559b60a-c2b7303a-cb18b9b4-4a253f92 has messages 1
2025-11-05T14:14:07.094531Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 read done: guid# 5559b60a-c2b7303a-cb18b9b4-4a253f92, partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), size# 178
2025-11-05T14:14:07.094571Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 response to read: guid# 5559b60a-c2b7303a-cb18b9b4-4a253f92
2025-11-05T14:14:07.094806Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Process answer. Aval parts: 0
2025-11-05T14:14:07.095075Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Got ReadResponse, serverBytesSize = 178, now ReadSizeBudget = 0, ReadSizeServerDelta = 8388430
2025-11-05T14:14:07.095193Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8388430
2025-11-05T14:14:07.096826Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 2 (0-0)
2025-11-05T14:14:07.096880Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 178 to budget
2025-11-05T14:14:07.096926Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 178, ReadSizeServerDelta = 8388430
2025-11-05T14:14:07.097178Z node 4 :PERSQUEUE DEBUG: Caching head blob in L1. Partition 0 offset 1 count 4 size 1600 actorID [4:7569244389242785200:2392]
2025-11-05T14:14:07.097247Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-11-05T14:14:07.097269Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 1496 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:07.097313Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.097336Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (0-0)
2025-11-05T14:14:07.097363Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 2, partNo: 0, Offset: 1 is stored on disk
2025-11-05T14:14:07.097393Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.097434Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 3, partNo: 0, Offset: 2 is stored on disk
2025-11-05T14:14:07.097455Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
>>> event from dataHandler: 2025-11-05T14:14:07.097490Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 4, partNo: 0, Offset: 3 is stored on disk
2025-11-05T14:14:07.097516Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
DataReceived { Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 Message { Data: ..20 bytes.. Information: { Offset: 0 ProducerId: "src_id" SeqNo: 1 CreateTime: 2025-11-05T14:14:07.030000Z WriteTime: 2025-11-05T14:14:07.033000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:45292", "logtype": "unknown", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } }
2025-11-05T14:14:07.097543Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic', Partition: 0, SeqNo: 5, partNo: 0, Offset: 4 is stored on disk
2025-11-05T14:14:07.097735Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.097759Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
>>> get 1 messages in this event
2025-11-05T14:14:07.097833Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 5  requestId: <none> cookie: 2
2025-11-05T14:14:07.097855Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] The application data is transferred to the client. Number of messages 1, size 20 bytes
2025-11-05T14:14:07.097847Z node 4 :PERSQUEUE DEBUG: PQ Cache (L2). Adding blob. Tablet '72075186224037892' partition 0 offset 1 partno 0 count 4 parts 0 size 1600
2025-11-05T14:14:07.097882Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 7  requestId: <none> cookie: 3
2025-11-05T14:14:07.097912Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 0 to budget
2025-11-05T14:14:07.097913Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 8  requestId: <none> cookie: 4
2025-11-05T14:14:07.097936Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 9  requestId: <none> cookie: 5
2025-11-05T14:14:07.098587Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic' requestId: <none>
2025-11-05T14:14:07.098622Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] got client message batch for topic 'rt3.dc1--test-topic' partition 0
2025-11-05T14:14:07.098063Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait for data done:  readOffset 1 EndOffset 1 newEndOffset 5 commitOffset 0 clientCommitOffset 0 cookie 2 readingFinished 0 firstRead 0
2025-11-05T14:14:07.098741Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] read cookie 2 Topic 'rt3.dc1--test-topic' partition 0 user user offset 1 count 4 size 1920 endOffset 5 max time lag 0ms effective offset 1
2025-11-05T14:14:07.098081Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.098782Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] read cookie 2 added 0 blobs, size 0 count 0 last offset 1, current partition end offset: 5
2025-11-05T14:14:07.098094Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) ready for read with readOffset 1 endOffset 5
2025-11-05T14:14:07.098795Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session got write response: sequence_numbers: 2 offsets: 1 already_written: false write_statistics { persist_duration_ms: 4 queued_in_partition_duration_ms: 4 }
2025-11-05T14:14:07.098140Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.098163Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), readOffset# 1, endOffset# 5, WTime# 1762352047033, sizeLag# 1600
2025-11-05T14:14:07.098865Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: acknoledged message 2
2025-11-05T14:14:07.098178Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.098179Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1TEvPartitionReady. Aval parts: 1
2025-11-05T14:14:07.098230Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 performing read request: guid# 72d4de4f-57f4e2ec-2f093b89-2aff2e37, from# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), count# 4, size# 1920, partitionsAsked# 1, maxTimeLag# 0ms
2025-11-05T14:14:07.098965Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Reading cookie 2. All data is from uncompacted head.
2025-11-05T14:14:07.098238Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.098285Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 READ FROM TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2)maxCount 4 maxSize 1920 maxTimeLagMs 0 readTimestampMs 0 readOffset 1 EndOffset 5 ClientCommitOffset 0 committedOffset 0 Guid 72d4de4f-57f4e2ec-2f093b89-2aff2e37
2025-11-05T14:14:07.099034Z node 4 :PERSQUEUE DEBUG: FormAnswer for 0 blobs
2025-11-05T14:14:07.098427Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { read_request { bytes_size: 178 } }
2025-11-05T14:14:07.098493Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got read request: guid# 117e4170-dc1f82b9-ce68f1b5-e8380eb9
2025-11-05T14:14:07.099162Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-11-05T14:14:07.099276Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session got write response: sequence_numbers: 3 offsets: 2 already_written: false write_statistics { persist_duration_ms: 4 queued_in_partition_duration_ms: 4 }
2025-11-05T14:14:07.099319Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: acknoledged message 3
2025-11-05T14:14:07.099556Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session got write response: sequence_numbers: 4 offsets: 3 already_written: false write_statistics { persist_duration_ms: 4 queued_in_partition_duration_ms: 4 }
2025-11-05T14:14:07.099598Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: acknoledged message 4
2025-11-05T14:14:07.099752Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) initDone 1 event { CmdReadResult { MaxOffset: 5 Result { Offset: 1 Data: "... 202 bytes ..." SourceId: "\000src_id" SeqNo: 2 WriteTimestampMS: 1762352047090 CreateTimestampMS: 1762352047030 UncompressedSize: 130 PartitionKey: "" ExplicitHash: "" } Result { Offset: 2 Data: "... 312 bytes ..." SourceId: "\000src_id" SeqNo: 3 WriteTimestampMS: 1762352047092 CreateTimestampMS: 1762352047031 UncompressedSize: 240 PartitionKey: "" ExplicitHash: "" } Result { Offset: 3 Data: "... 422 bytes ..." SourceId: "\000src_id" SeqNo: 4 WriteTimestampMS: 1762352047092 CreateTimestampMS: 1762352047031 UncompressedSize: 350 PartitionKey: "" ExplicitHash: "" } Result { Offset: 4 Data: "... 532 bytes ..." SourceId: "\000src_id" SeqNo: 5 WriteTimestampMS: 1762352047092 CreateTimestampMS: 1762352047031 UncompressedSize: 460 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 0 SizeLag: 18446744073709551580 RealReadOffset: 4 WaitQuotaTimeMs: 0 EndOffset: 5 StartOffset: 0 } Cookie: 1 }
2025-11-05T14:14:07.099819Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session got write response: sequence_numbers: 5 offsets: 4 already_written: false write_statistics { persist_duration_ms: 4 queued_in_partition_duration_ms: 3 }
2025-11-05T14:14:07.099858Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: acknoledged message 5
2025-11-05T14:14:07.099926Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) wait data in partition inited, cookie 3 from offset5
2025-11-05T14:14:07.099958Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 after read state TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2) EndOffset 5 ReadOffset 5 ReadGuid 72d4de4f-57f4e2ec-2f093b89-2aff2e37 has messages 1
2025-11-05T14:14:07.100062Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 read done: guid# 72d4de4f-57f4e2ec-2f093b89-2aff2e37, partition# TopicId: Topic rt3.dc1--test-topic in dc dc1 in database: Root, partition 0(assignId:2), size# 1562
2025-11-05T14:14:07.100094Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 response to read: guid# 72d4de4f-57f4e2ec-2f093b89-2aff2e37
2025-11-05T14:14:07.100282Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Process answer. Aval parts: 0
2025-11-05T14:14:07.100575Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Got ReadResponse, serverBytesSize = 1562, now ReadSizeBudget = 0, ReadSizeServerDelta = 8387046
2025-11-05T14:14:07.100709Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8387046
2025-11-05T14:14:07.101008Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 2 (1-4)
2025-11-05T14:14:07.101075Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 1562 to budget
2025-11-05T14:14:07.101149Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 1562, ReadSizeServerDelta = 8387046
2025-11-05T14:14:07.101465Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.101514Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-11-05T14:14:07.101499Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.101588Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (1-1)
2025-11-05T14:14:07.101638Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 0} (2-2)
2025-11-05T14:14:07.101739Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 1} (3-3)
2025-11-05T14:14:07.101783Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 2} (4-4)
2025-11-05T14:14:07.101730Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { read_request { bytes_size: 1562 } }
2025-11-05T14:14:07.101854Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got read request: guid# 6cd20260-f79d04cc-a1bf25a3-52042eb0
>>> event from dataHandler: DataReceived { Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 Message { Data: ..130 bytes.. Information: { Offset: 1 ProducerId: "src_id" SeqNo: 2 CreateTime: 2025-11-05T14:14:07.030000Z WriteTime: 2025-11-05T14:14:07.090000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "logtype": "unknown", "server": "ipv6:[::1]:45292", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } Message { Data: ..240 bytes.. Information: { Offset: 2 ProducerId: "src_id" SeqNo: 3 CreateTime: 2025-11-05T14:14:07.031000Z WriteTime: 2025-11-05T14:14:07.092000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:45292", "logtype": "unknown", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } Message { Data: ..350 bytes.. Information: { Offset: 3 ProducerId: "src_id" SeqNo: 4 CreateTime: 2025-11-05T14:14:07.031000Z WriteTime: 2025-11-05T14:14:07.092000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:45292", "logtype": "unknown", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } Message { Data: ..460 bytes.. Information: { Offset: 4 ProducerId: "src_id" SeqNo: 5 CreateTime: 2025-11-05T14:14:07.031000Z WriteTime: 2025-11-05T14:14:07.092000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:45292", "logtype": "unknown", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 1 Topic: "test-topic" Partition: 0 Database name: dc1 Database path: /Root Database id: account-dc1 } }
>>> get 4 messages in this event
2025-11-05T14:14:07.102267Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] The application data is transferred to the client. Number of messages 4, size 1180 bytes
2025-11-05T14:14:07.102305Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 0 to budget
2025-11-05T14:14:07.131709Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session will now close
2025-11-05T14:14:07.131812Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: aborting
2025-11-05T14:14:07.132730Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: gracefully shut down, all writes complete
>>> Writes to test-topic successful2025-11-05T14:14:07.132735Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session is aborting and will not restart
2025-11-05T14:14:07.132798Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0] Write session: destroy
2025-11-05T14:14:07.133020Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 2 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0 grpc read done: success: 0 data: 
2025-11-05T14:14:07.133048Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 2 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0 grpc read failed
2025-11-05T14:14:07.133557Z node 3 :PQ_WRITE_PROXY INFO: session v1 closed cookie: 2 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0
2025-11-05T14:14:07.133587Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 2 sessionId: src_id|fcb1b0e1-a6f0af56-295664c4-c6c1d7c_0 is DEAD
2025-11-05T14:14:07.133980Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037892 (partition=0) Received event: NActors::TEvents::TEvPoison
2025-11-05T14:14:07.134057Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: try to update token
2025-11-05T14:14:07.134122Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session: Do CDS request
2025-11-05T14:14:07.134154Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Start write session. Will connect to endpoint: localhost:20226
2025-11-05T14:14:07.134362Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] server disconnected, pipe [3:7569244409802246495:2567] destroyed
2025-11-05T14:14:07.134406Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] TPartition::DropOwner.
2025-11-05T14:14:07.134446Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.134482Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.138665Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: send init request: init_request { topic: "test-topic-mirrored-from-dc2" message_group_id: "src_id" }
2025-11-05T14:14:07.139016Z node 3 :PQ_WRITE_PROXY DEBUG: new grpc connection
2025-11-05T14:14:07.139044Z node 3 :PQ_WRITE_PROXY DEBUG: new session created cookie 3
2025-11-05T14:14:07.139416Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 3 sessionId:  grpc read done: success: 1 data: init_request { topic: "test-topic-mirrored-from-dc2" message_group_id: "src_id" }
2025-11-05T14:14:07.139577Z node 3 :PQ_WRITE_PROXY INFO: session request cookie: 3 topic: "test-topic-mirrored-from-dc2" message_group_id: "src_id" from ipv6:[::1]:45292
2025-11-05T14:14:07.139598Z node 3 :PQ_WRITE_PROXY INFO: write session:  cookie=3 sessionId= userAgent="pqv1 server" ip=ipv6:[::1]:45292 proto=v1  topic=test-topic-mirrored-from-dc2 durationSec=0
2025-11-05T14:14:07.139607Z node 3 :PQ_WRITE_PROXY INFO: init check schema
2025-11-05T14:14:07.141193Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 3 sessionId:  describe result for acl check
2025-11-05T14:14:07.141305Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper SelectQuery: --!syntax_v1
DECLARE $Hash AS Uint32; DECLARE $Topic AS Utf8; DECLARE $SourceId AS Utf8;
SELECT Partition, CreateTime, AccessTime, SeqNo FROM `/Root/PQ/SourceIdMeta2` WHERE Hash == $Hash AND Topic == $Topic AND SourceId == $SourceId;
2025-11-05T14:14:07.141319Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper UpdateQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;DECLARE $SeqNo AS Uint64;
UPSERT INTO `/Root/PQ/SourceIdMeta2` (Hash, Topic, SourceId, CreateTime, AccessTime, Partition, SeqNo) VALUES ($Hash, $Topic, $SourceId, $CreateTime, $AccessTime, $Partition, $SeqNo);
2025-11-05T14:14:07.141329Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper UpdateAccessTimeQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;
UPDATE `/Root/PQ/SourceIdMeta2` SET AccessTime = $AccessTime WHERE Hash = $Hash AND Topic = $Topic AND SourceId = $SourceId AND Partition = $Partition;
2025-11-05T14:14:07.141345Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246507:2571] (SourceId=src_id, PreferedPartition=(NULL)) StartKqpSession
2025-11-05T14:14:07.143681Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246507:2571] (SourceId=src_id, PreferedPartition=(NULL)) Select from the table
2025-11-05T14:14:07.148305Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246507:2571] (SourceId=src_id, PreferedPartition=(NULL)) RequestPQRB
2025-11-05T14:14:07.148523Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7569244409802246525:2571] connected; active server actors: 1
2025-11-05T14:14:07.148571Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246507:2571] (SourceId=src_id, PreferedPartition=(NULL)) Received partition 0 from PQRB for SourceId=src_id
2025-11-05T14:14:07.148588Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246507:2571] (SourceId=src_id, PreferedPartition=(NULL)) Update the table
2025-11-05T14:14:07.148869Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7569244409802246525:2571] disconnected; active server actors: 1
2025-11-05T14:14:07.148893Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7569244409802246525:2571] disconnected no session
2025-11-05T14:14:07.149642Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.149679Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.155664Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246507:2571] (SourceId=src_id, PreferedPartition=(NULL)) HandleUpdate PartitionPersisted=0 Status=SUCCESS
2025-11-05T14:14:07.155699Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246507:2571] (SourceId=src_id, PreferedPartition=(NULL)) ReplyResult: Partition=0, SeqNo=(NULL)
2025-11-05T14:14:07.155720Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246507:2571] (SourceId=src_id, PreferedPartition=(NULL)) Start idle
2025-11-05T14:14:07.155748Z node 3 :PQ_WRITE_PROXY DEBUG: ProceedPartition. session cookie: 3 sessionId:  partition: 0 expectedGeneration: (NULL)
2025-11-05T14:14:07.156334Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] server connected, pipe [3:7569244409802246537:2571], now have 1 active actors on pipe
2025-11-05T14:14:07.156452Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) TEvClientConnected Status OK, TabletId: 72075186224037894, NodeId 4, Generation: 1
2025-11-05T14:14:07.156599Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.156634Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.156744Z node 4 :PERSQUEUE INFO: new Cookie src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 generated for partition 0 topic 'rt3.dc1--test-topic-mirrored-from-dc2' owner src_id
2025-11-05T14:14:07.156787Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.156814Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.156857Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.156912Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::ReplyOwnerOk. Partition: 0
2025-11-05T14:14:07.157007Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:07.157340Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.157362Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.157462Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:07.157705Z node 3 :PQ_WRITE_PROXY INFO: session inited cookie: 3 partition: 0 MaxSeqNo: 0 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0
2025-11-05T14:14:07.158342Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1762352047158 BytesWritten: 0 MessagesWritten: 0 BytesWrittenCompressed: 0 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-11-05T14:14:07.158468Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session established. Init response: session_id: "src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0" topic: "test-topic-mirrored-from-dc2" cluster: "dc1" supported_codecs: CODEC_RAW supported_codecs: CODEC_GZIP supported_codecs: CODEC_LZOP
2025-11-05T14:14:07.158718Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write 1 messages with Id from 1 to 1
2025-11-05T14:14:07.158837Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: try to update token
2025-11-05T14:14:07.158868Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Send 1 message(s) (0 left), first sequence number is 1
2025-11-05T14:14:07.159076Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write 1 messages with Id from 2 to 2
2025-11-05T14:14:07.159134Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: try to update token
2025-11-05T14:14:07.159155Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Send 1 message(s) (0 left), first sequence number is 2
2025-11-05T14:14:07.159289Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write 1 messages with Id from 3 to 3
2025-11-05T14:14:07.159333Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: try to update token
2025-11-05T14:14:07.159351Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Send 1 message(s) (0 left), first sequence number is 3
2025-11-05T14:14:07.159390Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write 1 messages with Id from 4 to 4
2025-11-05T14:14:07.159420Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: try to update token
2025-11-05T14:14:07.159439Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Send 1 message(s) (0 left), first sequence number is 4
2025-11-05T14:14:07.159470Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write 1 messages with Id from 5 to 5
2025-11-05T14:14:07.159496Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: try to update token
2025-11-05T14:14:07.159512Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Send 1 message(s) (0 left), first sequence number is 5
2025-11-05T14:14:07.159793Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 3 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.159952Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: close. Timeout = 18446744073709551 ms
2025-11-05T14:14:07.159991Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 3 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.160200Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.160268Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 3 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.160438Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.160473Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 3 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.160512Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.160543Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.160588Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.160650Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.160668Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.160710Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-11-05T14:14:07.160759Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 1  requestId: <none> cookie: 2
2025-11-05T14:14:07.160738Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.160771Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 3 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.160927Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.161017Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.161043Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.161086Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 2  requestId: <none> cookie: 3
2025-11-05T14:14:07.161110Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.161170Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.161269Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.161294Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.161345Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.161327Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.161357Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.161450Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.161486Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.161800Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 1 partNo : 0 messageNo: 5 size 91 offset: -1
2025-11-05T14:14:07.161852Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 3  requestId: <none> cookie: 4
2025-11-05T14:14:07.161901Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.161913Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.161927Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.161946Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.161962Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 2 partNo : 0 messageNo: 6 size 202 offset: -1
2025-11-05T14:14:07.161977Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.161988Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 4  requestId: <none> cookie: 5
2025-11-05T14:14:07.162035Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.162054Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.162114Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.162071Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 1 partNo 0
2025-11-05T14:14:07.162079Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 3 partNo : 0 messageNo: 7 size 312 offset: -1
2025-11-05T14:14:07.162220Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.162241Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.162263Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 4 partNo : 0 messageNo: 8 size 422 offset: -1
2025-11-05T14:14:07.162424Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.162452Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.162487Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message topic: rt3.dc1--test-topic-mirrored-from-dc2 partition: 0 SourceId: '\0src_id' SeqNo: 5 partNo : 0 messageNo: 9 size 532 offset: -1
2025-11-05T14:14:07.163107Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 1 partNo 0 FormedBlobsCount 0 NewHead: Offset 0 PartNo 0 PackedSize 165 count 1 nextOffset 1 batches 1
2025-11-05T14:14:07.163169Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.163701Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Add new write blob: topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 compactOffset 0,1 HeadOffset 0 endOffset 0 curOffset 1 d0000000000_00000000000000000000_00000_0000000001_00000| size 153 WTime 1762352047163
2025-11-05T14:14:07.163880Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:07.163896Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.163919Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.163939Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.163963Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.163967Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough blob. Partition 0 offset 0 partNo 0 count 1 size 153
2025-11-05T14:14:07.167313Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.167355Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait for data done:  readOffset 0 EndOffset 0 newEndOffset 1 commitOffset 0 clientCommitOffset 0 cookie 1 readingFinished 0 firstRead 1
2025-11-05T14:14:07.167384Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) ready for read with readOffset 0 endOffset 1
2025-11-05T14:14:07.167457Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), readOffset# 0, endOffset# 1, WTime# 1762352045489, sizeLag# 153
2025-11-05T14:14:07.167475Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1TEvPartitionReady. Aval parts: 1
2025-11-05T14:14:07.167525Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 performing read request: guid# 23f83a57-ed157bb3-e8af31ab-98f21ae6, from# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), count# 1, size# 183, partitionsAsked# 1, maxTimeLag# 0ms
2025-11-05T14:14:07.167654Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 READ FROM TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3)maxCount 1 maxSize 183 maxTimeLagMs 0 readTimestampMs 0 readOffset 0 EndOffset 1 ClientCommitOffset 0 committedOffset 0 Guid 23f83a57-ed157bb3-e8af31ab-98f21ae6
2025-11-05T14:14:07.168032Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session got write response: sequence_numbers: 1 offsets: 0 already_written: false write_statistics { persist_duration_ms: 3 queued_in_partition_duration_ms: 1 }
2025-11-05T14:14:07.168081Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: acknoledged message 1
2025-11-05T14:14:07.166404Z node 4 :PERSQUEUE DEBUG: Caching head blob in L1. Partition 0 offset 0 count 1 size 153 actorID [4:7569244397832719989:2421]
2025-11-05T14:14:07.166506Z node 4 :PERSQUEUE DEBUG: PQ Cache (L2). Adding blob. Tablet '72075186224037894' partition 0 offset 0 partno 0 count 1 parts 0 size 153
2025-11-05T14:14:07.166520Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 98 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:07.166567Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.166632Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 1, partNo: 0, Offset: 0 is stored on disk
2025-11-05T14:14:07.166651Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 5  requestId: <none> cookie: 1
2025-11-05T14:14:07.166817Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-11-05T14:14:07.166850Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user send read request for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 1 rrg 0
2025-11-05T14:14:07.166877Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.166920Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.167022Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 2 partNo 0
2025-11-05T14:14:07.168040Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.168070Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.188012Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.188053Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.206681Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 2 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 276 count 1 nextOffset 2 batches 1
2025-11-05T14:14:07.206758Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 3 partNo 0
2025-11-05T14:14:07.206787Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 3 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 630 count 2 nextOffset 3 batches 1
2025-11-05T14:14:07.206803Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 4 partNo 0
2025-11-05T14:14:07.206817Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 4 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1094 count 3 nextOffset 4 batches 1
2025-11-05T14:14:07.206832Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob processing sourceId '\0src_id' seqNo 5 partNo 0
2025-11-05T14:14:07.206846Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 part blob complete sourceId '\0src_id' seqNo 5 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1668 count 4 nextOffset 5 batches 1
2025-11-05T14:14:07.206870Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.207387Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Add new write blob: topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 compactOffset 1,4 HeadOffset 0 endOffset 1 curOffset 5 d0000000000_00000000000000000001_00000_0000000004_00000| size 1600 WTime 1762352047202
2025-11-05T14:14:07.207552Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:07.207620Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough blob. Partition 0 offset 1 partNo 0 count 4 size 1600
2025-11-05T14:14:07.207874Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.207973Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] read cookie 0 Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user offset 0 count 1 size 1024000 endOffset 1 max time lag 0ms effective offset 0
2025-11-05T14:14:07.207999Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] read cookie 0 added 0 blobs, size 0 count 0 last offset 0, current partition end offset: 1
2025-11-05T14:14:07.208033Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Reading cookie 0. All data is from uncompacted head.
2025-11-05T14:14:07.208051Z node 4 :PERSQUEUE DEBUG: FormAnswer for 0 blobs
2025-11-05T14:14:07.208114Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] read cookie 1 Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user offset 0 count 1 size 183 endOffset 1 max time lag 0ms effective offset 0
2025-11-05T14:14:07.208132Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] read cookie 1 added 0 blobs, size 0 count 0 last offset 0, current partition end offset: 1
2025-11-05T14:14:07.208150Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Reading cookie 1. All data is from uncompacted head.
2025-11-05T14:14:07.208159Z node 4 :PERSQUEUE DEBUG: FormAnswer for 0 blobs
2025-11-05T14:14:07.208216Z node 4 :PERSQUEUE DEBUG: Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user readTimeStamp done, result 1762352047161 queuesize 0 startOffset 0
2025-11-05T14:14:07.208278Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:07.209289Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) initDone 1 event { CmdReadResult { MaxOffset: 1 Result { Offset: 0 Data: "... 91 bytes ..." SourceId: "\000src_id" SeqNo: 1 WriteTimestampMS: 1762352047161 CreateTimestampMS: 1762352047158 UncompressedSize: 20 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 0 SizeLag: 20 RealReadOffset: 0 WaitQuotaTimeMs: 0 EndOffset: 1 StartOffset: 0 } Cookie: 0 }
2025-11-05T14:14:07.209455Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait data in partition inited, cookie 2 from offset1
2025-11-05T14:14:07.209506Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 after read state TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) EndOffset 1 ReadOffset 1 ReadGuid 23f83a57-ed157bb3-e8af31ab-98f21ae6 has messages 1
2025-11-05T14:14:07.209719Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 read done: guid# 23f83a57-ed157bb3-e8af31ab-98f21ae6, partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), size# 178
2025-11-05T14:14:07.209780Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 response to read: guid# 23f83a57-ed157bb3-e8af31ab-98f21ae6
2025-11-05T14:14:07.210172Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Process answer. Aval parts: 0
2025-11-05T14:14:07.211028Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Got ReadResponse, serverBytesSize = 178, now ReadSizeBudget = 0, ReadSizeServerDelta = 8388430
2025-11-05T14:14:07.211153Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8388430
2025-11-05T14:14:07.211647Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 3 (0-0)
2025-11-05T14:14:07.211703Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 178 to budget
2025-11-05T14:14:07.211736Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 178, ReadSizeServerDelta = 8388430
2025-11-05T14:14:07.211956Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-11-05T14:14:07.212038Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (0-0)
>>> event from dataHandler: DataReceived { Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 Message { Data: ..20 bytes.. Information: { Offset: 0 ProducerId: "src_id" SeqNo: 1 CreateTime: 2025-11-05T14:14:07.158000Z WriteTime: 2025-11-05T14:14:07.161000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:45292", "logtype": "unknown", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } }
>>> get 1 messages in this event
2025-11-05T14:14:07.212315Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] The application data is transferred to the client. Number of messages 1, size 20 bytes
2025-11-05T14:14:07.212348Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 0 to budget
2025-11-05T14:14:07.212534Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { read_request { bytes_size: 178 } }
2025-11-05T14:14:07.212667Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got read request: guid# 44aeb87b-c349e8f5-63fd6f14-3cf29f41
2025-11-05T14:14:07.213466Z node 4 :PERSQUEUE DEBUG: Caching head blob in L1. Partition 0 offset 1 count 4 size 1600 actorID [4:7569244397832719989:2421]
2025-11-05T14:14:07.213613Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 1496 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:07.213628Z node 4 :PERSQUEUE DEBUG: PQ Cache (L2). Adding blob. Tablet '72075186224037894' partition 0 offset 1 partno 0 count 4 parts 0 size 1600
2025-11-05T14:14:07.213665Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.213712Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 6  requestId: <none> cookie: 2
2025-11-05T14:14:07.213721Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 2, partNo: 0, Offset: 1 is stored on disk
2025-11-05T14:14:07.213748Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.213779Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 7  requestId: <none> cookie: 3
2025-11-05T14:14:07.213795Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 3, partNo: 0, Offset: 2 is stored on disk
2025-11-05T14:14:07.213816Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.213848Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 4, partNo: 0, Offset: 3 is stored on disk
2025-11-05T14:14:07.213864Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.213884Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 8  requestId: <none> cookie: 4
2025-11-05T14:14:07.213888Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc2', Partition: 0, SeqNo: 5, partNo: 0, Offset: 4 is stored on disk
2025-11-05T14:14:07.213911Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 9  requestId: <none> cookie: 5
2025-11-05T14:14:07.214083Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.214108Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.214244Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.214311Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.214346Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.214380Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.215513Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait for data done:  readOffset 1 EndOffset 1 newEndOffset 5 commitOffset 0 clientCommitOffset 0 cookie 2 readingFinished 0 firstRead 0
2025-11-05T14:14:07.215570Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session got write response: sequence_numbers: 2 offsets: 1 already_written: false write_statistics { persist_duration_ms: 10 queued_in_partition_duration_ms: 38 }
2025-11-05T14:14:07.215547Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) ready for read with readOffset 1 endOffset 5
2025-11-05T14:14:07.215616Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: acknoledged message 2
2025-11-05T14:14:07.215619Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), readOffset# 1, endOffset# 5, WTime# 1762352047161, sizeLag# 1600
2025-11-05T14:14:07.215648Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1TEvPartitionReady. Aval parts: 1
2025-11-05T14:14:07.215698Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 performing read request: guid# 68dac5e2-cdb2f161-f3258b05-ab047f68, from# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), count# 4, size# 1920, partitionsAsked# 1, maxTimeLag# 0ms
2025-11-05T14:14:07.215785Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 READ FROM TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3)maxCount 4 maxSize 1920 maxTimeLagMs 0 readTimestampMs 0 readOffset 1 EndOffset 5 ClientCommitOffset 0 committedOffset 0 Guid 68dac5e2-cdb2f161-f3258b05-ab047f68
2025-11-05T14:14:07.215855Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session got write response: sequence_numbers: 3 offsets: 2 already_written: false write_statistics { persist_duration_ms: 10 queued_in_partition_duration_ms: 38 }
2025-11-05T14:14:07.215882Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: acknoledged message 3
2025-11-05T14:14:07.216068Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session got write response: sequence_numbers: 4 offsets: 3 already_written: false write_statistics { persist_duration_ms: 10 queued_in_partition_duration_ms: 38 }
2025-11-05T14:14:07.216103Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: acknoledged message 4
2025-11-05T14:14:07.216200Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc2' requestId: <none>
2025-11-05T14:14:07.216242Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0
2025-11-05T14:14:07.216283Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session got write response: sequence_numbers: 5 offsets: 4 already_written: false write_statistics { persist_duration_ms: 10 queued_in_partition_duration_ms: 38 }
2025-11-05T14:14:07.216312Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: acknoledged message 5
2025-11-05T14:14:07.216391Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] read cookie 2 Topic 'rt3.dc1--test-topic-mirrored-from-dc2' partition 0 user user offset 1 count 4 size 1920 endOffset 5 max time lag 0ms effective offset 1
2025-11-05T14:14:07.216427Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] read cookie 2 added 0 blobs, size 0 count 0 last offset 1, current partition end offset: 5
2025-11-05T14:14:07.216538Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Reading cookie 2. All data is from uncompacted head.
2025-11-05T14:14:07.216559Z node 4 :PERSQUEUE DEBUG: FormAnswer for 0 blobs
2025-11-05T14:14:07.216691Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc2' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-11-05T14:14:07.217251Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) initDone 1 event { CmdReadResult { MaxOffset: 5 Result { Offset: 1 Data: "... 202 bytes ..." SourceId: "\000src_id" SeqNo: 2 WriteTimestampMS: 1762352047166 CreateTimestampMS: 1762352047159 UncompressedSize: 130 PartitionKey: "" ExplicitHash: "" } Result { Offset: 2 Data: "... 312 bytes ..." SourceId: "\000src_id" SeqNo: 3 WriteTimestampMS: 1762352047202 CreateTimestampMS: 1762352047159 UncompressedSize: 240 PartitionKey: "" ExplicitHash: "" } Result { Offset: 3 Data: "... 422 bytes ..." SourceId: "\000src_id" SeqNo: 4 WriteTimestampMS: 1762352047202 CreateTimestampMS: 1762352047159 UncompressedSize: 350 PartitionKey: "" ExplicitHash: "" } Result { Offset: 4 Data: "... 532 bytes ..." SourceId: "\000src_id" SeqNo: 5 WriteTimestampMS: 1762352047202 CreateTimestampMS: 1762352047159 UncompressedSize: 460 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 0 SizeLag: 18446744073709551580 RealReadOffset: 4 WaitQuotaTimeMs: 0 EndOffset: 5 StartOffset: 0 } Cookie: 1 }
2025-11-05T14:14:07.217458Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) wait data in partition inited, cookie 3 from offset5
2025-11-05T14:14:07.217506Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 after read state TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3) EndOffset 5 ReadOffset 5 ReadGuid 68dac5e2-cdb2f161-f3258b05-ab047f68 has messages 1
2025-11-05T14:14:07.217643Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 read done: guid# 68dac5e2-cdb2f161-f3258b05-ab047f68, partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc2 in dc dc1 in database: Root, partition 0(assignId:3), size# 1562
2025-11-05T14:14:07.217686Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 response to read: guid# 68dac5e2-cdb2f161-f3258b05-ab047f68
2025-11-05T14:14:07.217904Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Process answer. Aval parts: 0
2025-11-05T14:14:07.220391Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Got ReadResponse, serverBytesSize = 1562, now ReadSizeBudget = 0, ReadSizeServerDelta = 8387046
2025-11-05T14:14:07.220530Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8387046
2025-11-05T14:14:07.220789Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 3 (1-4)
2025-11-05T14:14:07.220852Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 1562 to budget
2025-11-05T14:14:07.220885Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 1562, ReadSizeServerDelta = 8387046
2025-11-05T14:14:07.221115Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-11-05T14:14:07.221184Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (1-1)
2025-11-05T14:14:07.221222Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 0} (2-2)
2025-11-05T14:14:07.221260Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 1} (3-3)
2025-11-05T14:14:07.221292Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 2} (4-4)
>>> event from dataHandler: DataReceived { Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 Message { Data: ..130 bytes.. Information: { Offset: 1 ProducerId: "src_id" SeqNo: 2 CreateTime: 2025-11-05T14:14:07.159000Z WriteTime: 2025-11-05T14:14:07.166000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "_ip": "ipv6:[::1]:45292", "logtype": "unknown", "server": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } Message { Data: ..240 bytes.. Information: { Offset: 2 ProducerId: "src_id" SeqNo: 3 CreateTime: 2025-11-05T14:14:07.159000Z WriteTime: 2025-11-05T14:14:07.202000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:45292", "logtype": "unknown", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } Message { Data: ..350 bytes.. Information: { Offset: 3 ProducerId: "src_id" SeqNo: 4 CreateTime: 2025-11-05T14:14:07.159000Z WriteTime: 2025-11-05T14:14:07.202000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:45292", "logtype": "unknown", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } Message { Data: ..460 bytes.. Information: { Offset: 4 ProducerId: "src_id" SeqNo: 5 CreateTime: 2025-11-05T14:14:07.159000Z WriteTime: 2025-11-05T14:14:07.202000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:45292", "logtype": "unknown", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 2 Topic: "test-topic" Partition: 0 Database name: dc2 Database path: /Root Database id: account-dc2 } }
>>> get 4 messages in this event
2025-11-05T14:14:07.221677Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] The application data is transferred to the client. Number of messages 4, size 1180 bytes
2025-11-05T14:14:07.221656Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { read_request { bytes_size: 1562 } }
2025-11-05T14:14:07.221735Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 0 to budget
2025-11-05T14:14:07.221798Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got read request: guid# c6234ea8-5b0ce3ff-9a2ec5c2-4b19b20d
2025-11-05T14:14:07.250025Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.250070Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.260110Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session will now close
2025-11-05T14:14:07.260179Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: aborting
2025-11-05T14:14:07.260702Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: gracefully shut down, all writes complete
>>> Writes to test-topic-mirrored-from-dc2 successful
2025-11-05T14:14:07.260758Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0] Write session: destroy
2025-11-05T14:14:07.261226Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: try to update token
2025-11-05T14:14:07.261300Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session: Do CDS request
2025-11-05T14:14:07.261341Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Start write session. Will connect to endpoint: localhost:20226
2025-11-05T14:14:07.261422Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 3 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 grpc read done: success: 0 data: 
2025-11-05T14:14:07.261449Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 3 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 grpc read failed
2025-11-05T14:14:07.261477Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 3 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 grpc closed
2025-11-05T14:14:07.261495Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 3 sessionId: src_id|43698999-1ef54aff-66ad768e-ac6f0e45_0 is DEAD
2025-11-05T14:14:07.262398Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037894 (partition=0) Received event: NActors::TEvents::TEvPoison
2025-11-05T14:14:07.262707Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] server disconnected, pipe [3:7569244409802246537:2571] destroyed
2025-11-05T14:14:07.262768Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] TPartition::DropOwner.
2025-11-05T14:14:07.262819Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.262840Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.264412Z node 3 :PQ_WRITE_PROXY DEBUG: new grpc connection
2025-11-05T14:14:07.264439Z node 3 :PQ_WRITE_PROXY DEBUG: new session created cookie 4
2025-11-05T14:14:07.264494Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [] Write session: send init request: init_request { topic: "test-topic-mirrored-from-dc3" message_group_id: "src_id" }
2025-11-05T14:14:07.265194Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 4 sessionId:  grpc read done: success: 1 data: init_request { topic: "test-topic-mirrored-from-dc3" message_group_id: "src_id" }
2025-11-05T14:14:07.265345Z node 3 :PQ_WRITE_PROXY INFO: session request cookie: 4 topic: "test-topic-mirrored-from-dc3" message_group_id: "src_id" from ipv6:[::1]:45292
2025-11-05T14:14:07.265365Z node 3 :PQ_WRITE_PROXY INFO: write session:  cookie=4 sessionId= userAgent="pqv1 server" ip=ipv6:[::1]:45292 proto=v1  topic=test-topic-mirrored-from-dc3 durationSec=0
2025-11-05T14:14:07.265376Z node 3 :PQ_WRITE_PROXY INFO: init check schema
2025-11-05T14:14:07.266955Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 4 sessionId:  describe result for acl check
2025-11-05T14:14:07.267080Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper SelectQuery: --!syntax_v1
DECLARE $Hash AS Uint32; DECLARE $Topic AS Utf8; DECLARE $SourceId AS Utf8;
SELECT Partition, CreateTime, AccessTime, SeqNo FROM `/Root/PQ/SourceIdMeta2` WHERE Hash == $Hash AND Topic == $Topic AND SourceId == $SourceId;
2025-11-05T14:14:07.267096Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper UpdateQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;DECLARE $SeqNo AS Uint64;
UPSERT INTO `/Root/PQ/SourceIdMeta2` (Hash, Topic, SourceId, CreateTime, AccessTime, Partition, SeqNo) VALUES ($Hash, $Topic, $SourceId, $CreateTime, $AccessTime, $Partition, $SeqNo);
2025-11-05T14:14:07.267103Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TTableHelper UpdateAccessTimeQuery: --!syntax_v1
DECLARE $SourceId AS Utf8; DECLARE $Topic AS Utf8; DECLARE $Hash AS Uint32; DECLARE $Partition AS Uint32; DECLARE $CreateTime AS Uint64; DECLARE $AccessTime AS Uint64;
UPDATE `/Root/PQ/SourceIdMeta2` SET AccessTime = $AccessTime WHERE Hash = $Hash AND Topic = $Topic AND SourceId = $SourceId AND Partition = $Partition;
2025-11-05T14:14:07.267121Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246547:2576] (SourceId=src_id, PreferedPartition=(NULL)) StartKqpSession
2025-11-05T14:14:07.269676Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246547:2576] (SourceId=src_id, PreferedPartition=(NULL)) Select from the table
2025-11-05T14:14:07.274334Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246547:2576] (SourceId=src_id, PreferedPartition=(NULL)) RequestPQRB
2025-11-05T14:14:07.274566Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7569244409802246565:2576] connected; active server actors: 1
2025-11-05T14:14:07.274632Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246547:2576] (SourceId=src_id, PreferedPartition=(NULL)) Received partition 0 from PQRB for SourceId=src_id
2025-11-05T14:14:07.274660Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246547:2576] (SourceId=src_id, PreferedPartition=(NULL)) Update the table
2025-11-05T14:14:07.274901Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7569244409802246565:2576] disconnected; active server actors: 1
2025-11-05T14:14:07.274917Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7569244409802246565:2576] disconnected no session
2025-11-05T14:14:07.282501Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246547:2576] (SourceId=src_id, PreferedPartition=(NULL)) HandleUpdate PartitionPersisted=0 Status=SUCCESS
2025-11-05T14:14:07.282539Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246547:2576] (SourceId=src_id, PreferedPartition=(NULL)) ReplyResult: Partition=0, SeqNo=(NULL)
2025-11-05T14:14:07.282556Z node 3 :PQ_PARTITION_CHOOSER DEBUG: TPartitionChooser [3:7569244409802246547:2576] (SourceId=src_id, PreferedPartition=(NULL)) Start idle
2025-11-05T14:14:07.282584Z node 3 :PQ_WRITE_PROXY DEBUG: ProceedPartition. session cookie: 4 sessionId:  partition: 0 expectedGeneration: (NULL)
2025-11-05T14:14:07.283318Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] server connected, pipe [3:7569244409802246577:2576], now have 1 active actors on pipe
2025-11-05T14:14:07.283440Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) TEvClientConnected Status OK, TabletId: 72075186224037896, NodeId 4, Generation: 1
2025-11-05T14:14:07.283569Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.283600Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.283685Z node 4 :PERSQUEUE INFO: new Cookie src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 generated for partition 0 topic 'rt3.dc1--test-topic-mirrored-from-dc3' owner src_id
2025-11-05T14:14:07.283721Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.283748Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.283784Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.283823Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::ReplyOwnerOk. Partition: 0
2025-11-05T14:14:07.283882Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:07.284268Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.284289Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.284379Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:07.284687Z node 3 :PQ_WRITE_PROXY INFO: session inited cookie: 4 partition: 0 MaxSeqNo: 0 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0
2025-11-05T14:14:07.285491Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Counters: { Errors: 0 CurrentSessionLifetimeMs: 1762352047285 BytesWritten: 0 MessagesWritten: 0 BytesWrittenCompressed: 0 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-11-05T14:14:07.285618Z :INFO: [/Root] MessageGroupId [src_id] SessionId [] Write session established. Init response: session_id: "src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0" topic: "test-topic-mirrored-from-dc3" cluster: "dc1" supported_codecs: CODEC_RAW supported_codecs: CODEC_GZIP supported_codecs: CODEC_LZOP
2025-11-05T14:14:07.285924Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write 1 messages with Id from 1 to 1
2025-11-05T14:14:07.286052Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: try to update token
2025-11-05T14:14:07.286102Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Send 1 message(s) (0 left), first sequence number is 1
2025-11-05T14:14:07.286342Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write 1 messages with Id from 2 to 2
2025-11-05T14:14:07.286392Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: try to update token
2025-11-05T14:14:07.286418Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Send 1 message(s) (0 left), first sequence number is 2
2025-11-05T14:14:07.286467Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write 1 messages with Id from 3 to 3
2025-11-05T14:14:07.286504Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: try to update token
2025-11-05T14:14:07.286525Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Send 1 message(s) (0 left), first sequence number is 3
2025-11-05T14:14:07.286561Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write 1 messages with Id from 4 to 4
2025-11-05T14:14:07.286518Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 4 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.286600Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: try to update token
2025-11-05T14:14:07.286619Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Send 1 message(s) (0 left), first sequence number is 4
2025-11-05T14:14:07.286653Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write 1 messages with Id from 5 to 5
2025-11-05T14:14:07.286687Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: try to update token
2025-11-05T14:14:07.286705Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Send 1 message(s) (0 left), first sequence number is 5
2025-11-05T14:14:07.286732Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: close. Timeout = 18446744073709551 ms
2025-11-05T14:14:07.286826Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.287172Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.287204Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.287275Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-11-05T14:14:07.287366Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 4 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.287563Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.287707Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.287864Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.287889Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.287944Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 4 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.288116Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.288229Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 1 partNo : 0 messageNo: 1 size 91 offset: -1
2025-11-05T14:14:07.288290Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.288308Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.288301Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 4 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.288336Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.288351Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.288369Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.288398Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.288418Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.288421Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.288474Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 4 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 grpc read done: success: 1 data:  write_request[data omitted]
2025-11-05T14:14:07.288552Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 1 partNo 0
2025-11-05T14:14:07.288642Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.288773Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NKikimr::NPQ::TEvPartitionWriter::TEvWriteRequest
2025-11-05T14:14:07.288926Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.288956Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.288991Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.289003Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.289576Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 1 partNo 0 FormedBlobsCount 0 NewHead: Offset 0 PartNo 0 PackedSize 165 count 1 nextOffset 1 batches 1
2025-11-05T14:14:07.289632Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.290148Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Add new write blob: topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 compactOffset 0,1 HeadOffset 0 endOffset 0 curOffset 1 d0000000000_00000000000000000000_00000_0000000001_00000| size 153 WTime 1762352047289
2025-11-05T14:14:07.290312Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:07.290386Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 2  requestId: <none> cookie: 2
2025-11-05T14:14:07.290402Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough blob. Partition 0 offset 0 partNo 0 count 1 size 153
2025-11-05T14:14:07.290446Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 3  requestId: <none> cookie: 3
2025-11-05T14:14:07.290466Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 4  requestId: <none> cookie: 4
2025-11-05T14:14:07.290487Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 5  requestId: <none> cookie: 5
2025-11-05T14:14:07.290578Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.290646Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.290698Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.290745Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.291015Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.291048Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.291097Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 2 partNo : 0 messageNo: 6 size 202 offset: -1
2025-11-05T14:14:07.291158Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.291173Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.291174Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.291192Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 3 partNo : 0 messageNo: 7 size 312 offset: -1
2025-11-05T14:14:07.291218Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.291225Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.291240Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.291260Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 4 partNo : 0 messageNo: 8 size 422 offset: -1
2025-11-05T14:14:07.291304Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.291315Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.291335Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message topic: rt3.dc1--test-topic-mirrored-from-dc3 partition: 0 SourceId: '\0src_id' SeqNo: 5 partNo : 0 messageNo: 9 size 532 offset: -1
2025-11-05T14:14:07.291342Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.291375Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Can't process txs
2025-11-05T14:14:07.293169Z node 4 :PERSQUEUE DEBUG: Caching head blob in L1. Partition 0 offset 0 count 1 size 153 actorID [4:7569244402127687416:2439]
2025-11-05T14:14:07.293275Z node 4 :PERSQUEUE DEBUG: PQ Cache (L2). Adding blob. Tablet '72075186224037896' partition 0 offset 0 partno 0 count 1 parts 0 size 153
2025-11-05T14:14:07.293331Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 98 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:07.293381Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.293464Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 1, partNo: 0, Offset: 0 is stored on disk
2025-11-05T14:14:07.293683Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user readTimeStamp for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 0 rrg 0
2025-11-05T14:14:07.293720Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user send read request for offset 0 initiated  queuesize 0 startOffset 0 ReadingTimestamp 1 rrg 0
2025-11-05T14:14:07.293758Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.293806Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.293826Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait for data done:  readOffset 0 EndOffset 0 newEndOffset 1 commitOffset 0 clientCommitOffset 0 cookie 1 readingFinished 0 firstRead 1
2025-11-05T14:14:07.293861Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) ready for read with readOffset 0 endOffset 1
2025-11-05T14:14:07.293908Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 2 partNo 0
2025-11-05T14:14:07.293936Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), readOffset# 0, endOffset# 1, WTime# 1762352046242, sizeLag# 153
2025-11-05T14:14:07.293958Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1TEvPartitionReady. Aval parts: 1
2025-11-05T14:14:07.294021Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 performing read request: guid# a40cfdf5-c7d89fea-7befab56-28fc9e4a, from# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), count# 1, size# 183, partitionsAsked# 1, maxTimeLag# 0ms
2025-11-05T14:14:07.294112Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 READ FROM TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1)maxCount 1 maxSize 183 maxTimeLagMs 0 readTimestampMs 0 readOffset 0 EndOffset 1 ClientCommitOffset 0 committedOffset 0 Guid a40cfdf5-c7d89fea-7befab56-28fc9e4a
2025-11-05T14:14:07.294628Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.294435Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 1  requestId: <none> cookie: 1
2025-11-05T14:14:07.294582Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.294606Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.295552Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session got write response: sequence_numbers: 1 offsets: 0 already_written: false write_statistics { persist_duration_ms: 3 }
2025-11-05T14:14:07.295602Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: acknoledged message 1
2025-11-05T14:14:07.303388Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.303418Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.334810Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 2 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 276 count 1 nextOffset 2 batches 1
2025-11-05T14:14:07.334899Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 3 partNo 0
2025-11-05T14:14:07.334939Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 3 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 630 count 2 nextOffset 3 batches 1
2025-11-05T14:14:07.334963Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 4 partNo 0
2025-11-05T14:14:07.334987Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 4 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1094 count 3 nextOffset 4 batches 1
2025-11-05T14:14:07.335016Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob processing sourceId '\0src_id' seqNo 5 partNo 0
2025-11-05T14:14:07.335043Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 part blob complete sourceId '\0src_id' seqNo 5 partNo 0 FormedBlobsCount 0 NewHead: Offset 1 PartNo 0 PackedSize 1668 count 4 nextOffset 5 batches 1
2025-11-05T14:14:07.335083Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.335618Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Add new write blob: topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 compactOffset 1,4 HeadOffset 0 endOffset 1 curOffset 5 d0000000000_00000000000000000001_00000_0000000004_00000| size 1600 WTime 1762352047335
2025-11-05T14:14:07.335808Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough write request to KV
2025-11-05T14:14:07.335899Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] read cookie 0 Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user offset 0 count 1 size 1024000 endOffset 1 max time lag 0ms effective offset 0
2025-11-05T14:14:07.335910Z node 4 :PERSQUEUE DEBUG: CacheProxy. Passthrough blob. Partition 0 offset 1 partNo 0 count 4 size 1600
2025-11-05T14:14:07.335932Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] read cookie 0 added 0 blobs, size 0 count 0 last offset 0, current partition end offset: 1
2025-11-05T14:14:07.335966Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Reading cookie 0. All data is from uncompacted head.
2025-11-05T14:14:07.335988Z node 4 :PERSQUEUE DEBUG: FormAnswer for 0 blobs
2025-11-05T14:14:07.336048Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] read cookie 1 Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user offset 0 count 1 size 183 endOffset 1 max time lag 0ms effective offset 0
2025-11-05T14:14:07.336073Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] read cookie 1 added 0 blobs, size 0 count 0 last offset 0, current partition end offset: 1
2025-11-05T14:14:07.336092Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Reading cookie 1. All data is from uncompacted head.
2025-11-05T14:14:07.336101Z node 4 :PERSQUEUE DEBUG: FormAnswer for 0 blobs
2025-11-05T14:14:07.336151Z node 4 :PERSQUEUE DEBUG: Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user readTimeStamp done, result 1762352047288 queuesize 0 startOffset 0
2025-11-05T14:14:07.336172Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 0
2025-11-05T14:14:07.336702Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) initDone 1 event { CmdReadResult { MaxOffset: 1 Result { Offset: 0 Data: "... 91 bytes ..." SourceId: "\000src_id" SeqNo: 1 WriteTimestampMS: 1762352047288 CreateTimestampMS: 1762352047285 UncompressedSize: 20 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 0 SizeLag: 20 RealReadOffset: 0 WaitQuotaTimeMs: 0 EndOffset: 1 StartOffset: 0 } Cookie: 0 }
2025-11-05T14:14:07.336831Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait data in partition inited, cookie 2 from offset1
2025-11-05T14:14:07.336868Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 after read state TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) EndOffset 1 ReadOffset 1 ReadGuid a40cfdf5-c7d89fea-7befab56-28fc9e4a has messages 1
2025-11-05T14:14:07.337001Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 read done: guid# a40cfdf5-c7d89fea-7befab56-28fc9e4a, partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), size# 180
2025-11-05T14:14:07.337037Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 response to read: guid# a40cfdf5-c7d89fea-7befab56-28fc9e4a
2025-11-05T14:14:07.337304Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Process answer. Aval parts: 0
2025-11-05T14:14:07.340856Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Got ReadResponse, serverBytesSize = 180, now ReadSizeBudget = 0, ReadSizeServerDelta = 8388428
2025-11-05T14:14:07.340956Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8388428
2025-11-05T14:14:07.341299Z node 4 :PERSQUEUE DEBUG: Caching head blob in L1. Partition 0 offset 1 count 4 size 1600 actorID [4:7569244402127687416:2439]
2025-11-05T14:14:07.341420Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 1 (0-0)
2025-11-05T14:14:07.341430Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::HandleWriteResponse writeNewSize# 1496 WriteNewSizeFromSupportivePartitions# 0
2025-11-05T14:14:07.341477Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 180 to budget
2025-11-05T14:14:07.341461Z node 4 :PERSQUEUE DEBUG: PQ Cache (L2). Adding blob. Tablet '72075186224037896' partition 0 offset 1 partno 0 count 4 parts 0 size 1600
2025-11-05T14:14:07.341519Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 180, ReadSizeServerDelta = 8388428
2025-11-05T14:14:07.341493Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.341561Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 2, partNo: 0, Offset: 1 is stored on disk
2025-11-05T14:14:07.341584Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 6  requestId: <none> cookie: 2
2025-11-05T14:14:07.341592Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.341614Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 3, partNo: 0, Offset: 2 is stored on disk
2025-11-05T14:14:07.341630Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.341632Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 7  requestId: <none> cookie: 3
2025-11-05T14:14:07.341652Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 8  requestId: <none> cookie: 4
2025-11-05T14:14:07.341654Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 4, partNo: 0, Offset: 3 is stored on disk
2025-11-05T14:14:07.341674Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::ReplyWrite. Partition: 0
2025-11-05T14:14:07.341706Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Answering for message sourceid: '\0src_id', Topic: 'rt3.dc1--test-topic-mirrored-from-dc3', Partition: 0, SeqNo: 5, partNo: 0, Offset: 4 is stored on disk
2025-11-05T14:14:07.341771Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-11-05T14:14:07.341835Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (0-0)
2025-11-05T14:14:07.341858Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.341882Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.341950Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 9  requestId: <none> cookie: 5
>>> event from dataHandler: DataReceived { Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 Message { Data: ..20 bytes.. Information: { Offset: 0 ProducerId: "src_id" SeqNo: 1 CreateTime: 2025-11-05T14:14:07.285000Z WriteTime: 2025-11-05T14:14:07.288000Z MessageGroupId: "src_id" Meta: { "_ip": "ipv6:[::1]:45292", "server": "ipv6:[::1]:45292", "logtype": "unknown", "ident": "unknown" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } }
>>> get 1 messages in this event
2025-11-05T14:14:07.342125Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] The application data is transferred to the client. Number of messages 1, size 20 bytes
2025-11-05T14:14:07.342173Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 0 to budget
2025-11-05T14:14:07.342200Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { read_request { bytes_size: 180 } }
2025-11-05T14:14:07.342359Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got read request: guid# 8af1f7e5-686aba0-8514e71-84a08c38
2025-11-05T14:14:07.342617Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.343204Z node 4 :PERSQUEUE DEBUG: Handle TEvRequest topic: 'rt3.dc1--test-topic-mirrored-from-dc3' requestId: <none>
2025-11-05T14:14:07.342651Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait for data done:  readOffset 1 EndOffset 1 newEndOffset 5 commitOffset 0 clientCommitOffset 0 cookie 2 readingFinished 0 firstRead 0
2025-11-05T14:14:07.343259Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session got write response: sequence_numbers: 2 offsets: 1 already_written: false write_statistics { persist_duration_ms: 6 queued_in_partition_duration_ms: 44 }
2025-11-05T14:14:07.343247Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] got client message batch for topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0
2025-11-05T14:14:07.342680Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) ready for read with readOffset 1 endOffset 5
2025-11-05T14:14:07.342684Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.343307Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: acknoledged message 2
2025-11-05T14:14:07.342710Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.342723Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 partition ready for read: partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), readOffset# 1, endOffset# 5, WTime# 1762352047288, sizeLag# 1600
2025-11-05T14:14:07.342738Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1TEvPartitionReady. Aval parts: 1
2025-11-05T14:14:07.342745Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::IEventHandle
2025-11-05T14:14:07.342780Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 performing read request: guid# 7336ef30-1cb95a39-63e4368e-b3eee50f, from# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), count# 4, size# 1920, partitionsAsked# 1, maxTimeLag# 0ms
2025-11-05T14:14:07.342831Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 READ FROM TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1)maxCount 4 maxSize 1920 maxTimeLagMs 0 readTimestampMs 0 readOffset 1 EndOffset 5 ClientCommitOffset 0 committedOffset 0 Guid 7336ef30-1cb95a39-63e4368e-b3eee50f
2025-11-05T14:14:07.343367Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] read cookie 2 Topic 'rt3.dc1--test-topic-mirrored-from-dc3' partition 0 user user offset 1 count 4 size 1920 endOffset 5 max time lag 0ms effective offset 1
2025-11-05T14:14:07.343398Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] read cookie 2 added 0 blobs, size 0 count 0 last offset 1, current partition end offset: 5
2025-11-05T14:14:07.343854Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session got write response: sequence_numbers: 3 offsets: 2 already_written: false write_statistics { persist_duration_ms: 6 queued_in_partition_duration_ms: 44 }
2025-11-05T14:14:07.343527Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Reading cookie 2. All data is from uncompacted head.
2025-11-05T14:14:07.343894Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: acknoledged message 3
2025-11-05T14:14:07.343559Z node 4 :PERSQUEUE DEBUG: FormAnswer for 0 blobs
2025-11-05T14:14:07.343661Z node 4 :PERSQUEUE DEBUG: Answer ok topic: 'rt3.dc1--test-topic-mirrored-from-dc3' partition: 0 messageNo: 0  requestId: <none> cookie: 1
2025-11-05T14:14:07.344248Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session got write response: sequence_numbers: 4 offsets: 3 already_written: false write_statistics { persist_duration_ms: 6 queued_in_partition_duration_ms: 44 }
2025-11-05T14:14:07.344284Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: acknoledged message 4
2025-11-05T14:14:07.344547Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session got write response: sequence_numbers: 5 offsets: 4 already_written: false write_statistics { persist_duration_ms: 6 queued_in_partition_duration_ms: 44 }
2025-11-05T14:14:07.344598Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: acknoledged message 5
2025-11-05T14:14:07.345493Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Got ReadResponse, serverBytesSize = 1568, now ReadSizeBudget = 0, ReadSizeServerDelta = 8387040
2025-11-05T14:14:07.345611Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 0, ReadSizeServerDelta = 8387040
2025-11-05T14:14:07.345829Z :DEBUG: [/Root] Decompression task done. Partition/PartitionSessionId: 1 (1-4)
2025-11-05T14:14:07.345887Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 1568 to budget
2025-11-05T14:14:07.345919Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] In ContinueReadingDataImpl, ReadSizeBudget = 1568, ReadSizeServerDelta = 8387040
2025-11-05T14:14:07.346168Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] After sending read request: ReadSizeBudget = 0, ReadSizeServerDelta = 8388608
2025-11-05T14:14:07.346276Z :DEBUG: [/Root] Take Data. Partition 0. Read: {0, 0} (1-1)
2025-11-05T14:14:07.346316Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 0} (2-2)
2025-11-05T14:14:07.346356Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 1} (3-3)
2025-11-05T14:14:07.346394Z :DEBUG: [/Root] Take Data. Partition 0. Read: {1, 2} (4-4)
>>> event from dataHandler: DataReceived { Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 Message { Data: ..130 bytes.. Information: { Offset: 1 ProducerId: "src_id" SeqNo: 2 CreateTime: 2025-11-05T14:14:07.286000Z WriteTime: 2025-11-05T14:14:07.293000Z MessageGroupId: "src_id" Meta: { "ident": "unknown", "server": "ipv6:[::1]:45292", "logtype": "unknown", "_ip": "ipv6:[::1]:45292" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } Message { Data: ..240 bytes.. Information: { Offset: 2 ProducerId: "src_id" SeqNo: 3 CreateTime: 2025-11-05T14:14:07.286000Z WriteTime: 2025-11-05T14:14:07.334000Z MessageGroupId: "src_id" Meta: { "_ip": "ipv6:[::1]:45292", "logtype": "unknown", "server": "ipv6:[::1]:45292", "ident": "unknown" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } Message { Data: ..350 bytes.. Information: { Offset: 3 ProducerId: "src_id" SeqNo: 4 CreateTime: 2025-11-05T14:14:07.286000Z WriteTime: 2025-11-05T14:14:07.334000Z MessageGroupId: "src_id" Meta: { "_ip": "ipv6:[::1]:45292", "logtype": "unknown", "server": "ipv6:[::1]:45292", "ident": "unknown" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } Message { Data: ..460 bytes.. Information: { Offset: 4 ProducerId: "src_id" SeqNo: 5 CreateTime: 2025-11-05T14:14:07.286000Z WriteTime: 2025-11-05T14:14:07.334000Z MessageGroupId: "src_id" Meta: { "_ip": "ipv6:[::1]:45292", "logtype": "unknown", "server": "ipv6:[::1]:45292", "ident": "unknown" } MessageMeta: { } } Partition session id: 3 Topic: "test-topic" Partition: 0 Database name: dc3 Database path: /Root Database id: account-dc3 } }
>>> get 4 messages in this event
2025-11-05T14:14:07.347525Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] The application data is transferred to the client. Number of messages 4, size 1180 bytes
2025-11-05T14:14:07.347564Z :DEBUG: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] [] Returning serverBytesSize = 0 to budget
2025-11-05T14:14:07.344593Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) initDone 1 event { CmdReadResult { MaxOffset: 5 Result { Offset: 1 Data: "... 202 bytes ..." SourceId: "\000src_id" SeqNo: 2 WriteTimestampMS: 1762352047293 CreateTimestampMS: 1762352047286 UncompressedSize: 130 PartitionKey: "" ExplicitHash: "" } Result { Offset: 2 Data: "... 312 bytes ..." SourceId: "\000src_id" SeqNo: 3 WriteTimestampMS: 1762352047334 CreateTimestampMS: 1762352047286 UncompressedSize: 240 PartitionKey: "" ExplicitHash: "" } Result { Offset: 3 Data: "... 422 bytes ..." SourceId: "\000src_id" SeqNo: 4 WriteTimestampMS: 1762352047334 CreateTimestampMS: 1762352047286 UncompressedSize: 350 PartitionKey: "" ExplicitHash: "" } Result { Offset: 4 Data: "... 532 bytes ..." SourceId: "\000src_id" SeqNo: 5 WriteTimestampMS: 1762352047334 CreateTimestampMS: 1762352047286 UncompressedSize: 460 PartitionKey: "" ExplicitHash: "" } BlobsFromDisk: 0 BlobsFromCache: 0 SizeLag: 18446744073709551580 RealReadOffset: 4 WaitQuotaTimeMs: 0 EndOffset: 5 StartOffset: 0 } Cookie: 1 }
2025-11-05T14:14:07.344817Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) wait data in partition inited, cookie 3 from offset5
2025-11-05T14:14:07.344854Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 after read state TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1) EndOffset 5 ReadOffset 5 ReadGuid 7336ef30-1cb95a39-63e4368e-b3eee50f has messages 1
2025-11-05T14:14:07.344988Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 read done: guid# 7336ef30-1cb95a39-63e4368e-b3eee50f, partition# TopicId: Topic rt3.dc1--test-topic-mirrored-from-dc3 in dc dc1 in database: Root, partition 0(assignId:1), size# 1568
2025-11-05T14:14:07.345021Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 response to read: guid# 7336ef30-1cb95a39-63e4368e-b3eee50f
2025-11-05T14:14:07.345230Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 Process answer. Aval parts: 0
2025-11-05T14:14:07.347253Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 1, data# { read_request { bytes_size: 1568 } }
2025-11-05T14:14:07.347364Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 got read request: guid# 9a3c1edc-ca6f7309-c612e717-1a6c9da3
2025-11-05T14:14:07.350267Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.350303Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.386885Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session will now close
2025-11-05T14:14:07.386983Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: aborting
2025-11-05T14:14:07.387463Z :INFO: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: gracefully shut down, all writes complete
>>> Writes to test-topic-mirrored-from-dc3 successful
2025-11-05T14:14:07.387515Z :DEBUG: [/Root] MessageGroupId [src_id] SessionId [src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0] Write session: destroy
2025-11-05T14:14:07.387688Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] Closing read session. Close timeout: 18446744073709.551615s
2025-11-05T14:14:07.387781Z :INFO: [/Root] Read/commit by partition streams (cluster:topic:partition:stream-id:read-offset:committed-offset): -:test-topic-mirrored-from-dc3:0:3:4:0 -:test-topic-mirrored-from-dc2:0:2:4:0 -:test-topic:0:1:4:0
2025-11-05T14:14:07.387828Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] Counters: { Errors: 0 CurrentSessionLifetimeMs: 423 BytesRead: 3600 MessagesRead: 15 BytesReadCompressed: 3600 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-11-05T14:14:07.388250Z node 3 :PQ_WRITE_PROXY DEBUG: session v1 cookie: 4 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 grpc read done: success: 0 data: 
2025-11-05T14:14:07.388352Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] Closing read session. Close timeout: 0.000000s
2025-11-05T14:14:07.388288Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 4 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 grpc read failed
2025-11-05T14:14:07.388319Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 4 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 grpc closed
2025-11-05T14:14:07.388405Z :INFO: [/Root] Read/commit by partition streams (cluster:topic:partition:stream-id:read-offset:committed-offset): -:test-topic-mirrored-from-dc3:0:3:4:0 -:test-topic-mirrored-from-dc2:0:2:4:0 -:test-topic:0:1:4:0
2025-11-05T14:14:07.388339Z node 3 :PQ_WRITE_PROXY INFO: session v1 cookie: 4 sessionId: src_id|e61ec007-4dd7a3e9-f372d683-d230c6c2_0 is DEAD
2025-11-05T14:14:07.388444Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] Counters: { Errors: 0 CurrentSessionLifetimeMs: 424 BytesRead: 3600 MessagesRead: 15 BytesReadCompressed: 3600 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-11-05T14:14:07.388480Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] Closing read session. Close timeout: 0.000000s
2025-11-05T14:14:07.388521Z :INFO: [/Root] Read/commit by partition streams (cluster:topic:partition:stream-id:read-offset:committed-offset): -:test-topic-mirrored-from-dc3:0:3:4:0 -:test-topic-mirrored-from-dc2:0:2:4:0 -:test-topic:0:1:4:0
2025-11-05T14:14:07.388554Z :INFO: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] Counters: { Errors: 0 CurrentSessionLifetimeMs: 424 BytesRead: 3600 MessagesRead: 15 BytesReadCompressed: 3600 BytesInflightUncompressed: 0 BytesInflightCompressed: 0 BytesInflightTotal: 0 MessagesInflight: 0 }
2025-11-05T14:14:07.388643Z :NOTICE: [/Root] [/Root] [2915aee6-23128419-5560eaf4-496e0caf] Aborting read session. Description: SessionClosed { Status: ABORTED Issues: "<main>: Error: Aborted " }
2025-11-05T14:14:07.388728Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.388767Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.388799Z node 3 :PQ_READ_PROXY DEBUG: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read done: success# 0, data# {  }
2025-11-05T14:14:07.388820Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc read failed
2025-11-05T14:14:07.388853Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 grpc closed
2025-11-05T14:14:07.388953Z node 3 :PQ_READ_PROXY INFO: session cookie 1 consumer shared/user session shared/user_3_1_5731498468557581848_v1 is DEAD
2025-11-05T14:14:07.389267Z node 3 :PQ_WRITE_PROXY DEBUG: TPartitionWriter 72075186224037896 (partition=0) Received event: NActors::TEvents::TEvPoison
2025-11-05T14:14:07.389783Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] server disconnected, pipe [3:7569244409802246577:2576] destroyed
2025-11-05T14:14:07.389821Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] Destroy direct read session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:07.389862Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896] server disconnected, pipe [3:7569244405507279156:2562] destroyed
2025-11-05T14:14:07.390008Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] Destroy direct read session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:07.390032Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892] server disconnected, pipe [3:7569244405507279162:2565] destroyed
2025-11-05T14:14:07.390129Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] Destroy direct read session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:07.390168Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894] server disconnected, pipe [3:7569244405507279163:2566] destroyed
2025-11-05T14:14:07.390177Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] TPartition::DropOwner.
2025-11-05T14:14:07.390216Z node 4 :PQ_READ_PROXY DEBUG: Direct read cache: server session deregistered: shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:07.390222Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.390243Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.390244Z node 4 :PQ_READ_PROXY DEBUG: Direct read cache: server session deregistered: shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:07.390259Z node 4 :PQ_READ_PROXY DEBUG: Direct read cache: server session deregistered: shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:07.391143Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7569244405507279149:2559] disconnected; active server actors: 1
2025-11-05T14:14:07.391159Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037893][rt3.dc1--test-topic] pipe [3:7569244405507279150:2559] disconnected; active server actors: 1
2025-11-05T14:14:07.391174Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: [72075186224037897][rt3.dc1--test-topic-mirrored-from-dc3] pipe [3:7569244405507279149:2559] client user disconnected session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:07.391197Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: [72075186224037893][rt3.dc1--test-topic] pipe [3:7569244405507279150:2559] client user disconnected session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:07.391245Z node 3 :PERSQUEUE_READ_BALANCER INFO: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7569244405507279151:2559] disconnected; active server actors: 1
2025-11-05T14:14:07.391258Z node 3 :PERSQUEUE_READ_BALANCER NOTICE: [72075186224037895][rt3.dc1--test-topic-mirrored-from-dc2] pipe [3:7569244405507279151:2559] client user disconnected session shared/user_3_1_5731498468557581848_v1
2025-11-05T14:14:07.403613Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.403651Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.450616Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.450640Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.489051Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.489093Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037892, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.503891Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.503929Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037894, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.550998Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state before ContinueProcessTxsAndUserActs: 0
2025-11-05T14:14:07.551042Z node 4 :PERSQUEUE DEBUG: [PQ: 72075186224037896, Partition: 0, State: StateIdle] Batching state after ContinueProcessTxsAndUserActs: 1
2025-11-05T14:14:07.756947Z node 3 :KQP_COMPUTE WARN: SelfId: [3:7569244409802246623:2586], TxId: 281474976715704, task: 1, CA Id [3:7569244409802246621:2586]. Got EvDeliveryProblem, TabletId: 72075186224037890, NotDelivered: 0
2025-11-05T14:14:07.793248Z node 3 :KQP_COMPUTE WARN: SelfId: [3:7569244409802246623:2586], TxId: 281474976715704, task: 1, CA Id [3:7569244409802246621:2586]. Got EvDeliveryProblem, TabletId: 72075186224037890, NotDelivered: 1
2025-11-05T14:14:07.837744Z node 3 :KQP_COMPUTE WARN: SelfId: [3:7569244409802246623:2586], TxId: 281474976715704, task: 1, CA Id [3:7569244409802246621:2586]. Got EvDeliveryProblem, TabletId: 72075186224037890, NotDelivered: 1
2025-11-05T14:14:07.878590Z node 3 :KQP_COMPUTE WARN: SelfId: [3:7569244409802246623:2586], TxId: 281474976715704, task: 1, CA Id [3:7569244409802246621:2586]. Got EvDeliveryProblem, TabletId: 72075186224037890, NotDelivered: 1
2025-11-05T14:14:07.982028Z node 3 :KQP_COMPUTE WARN: SelfId: [3:7569244409802246623:2586], TxId: 281474976715704, task: 1, CA Id [3:7569244409802246621:2586]. Got EvDeliveryProblem, TabletId: 72075186224037890, NotDelivered: 1
2025-11-05T14:14:08.016351Z node 3 :KQP_EXECUTER WARN: [ShardsResolver] TxId: 281474976715705. Failed to resolve tablet: 72075186224037890 after several retries.
2025-11-05T14:14:08.016487Z node 3 :KQP_EXECUTER WARN: ActorId: [3:7569244409802246637:2580] TxId: 281474976715705. Ctx: { TraceId: 01k9a5w9ce9q8vhbht8m1dxs2e, Database: /Root, DatabaseId: /Root, SessionId: ydb://session/3?node_id=3&id=N2M4YmRlMTItZmZmMTE1NjAtM2I2Y2Y5MzUtNjI4YjEzZGI=, CurrentExecutionId: , CustomerSuppliedId: , PoolId: }. Shards nodes resolve failed, status: UNAVAILABLE, issues: <main>: Error: Failed to resolve tablet: 72075186224037890 after several retries.
2025-11-05T14:14:08.016737Z node 3 :KQP_SESSION WARN: SessionId: ydb://session/3?node_id=3&id=N2M4YmRlMTItZmZmMTE1NjAtM2I2Y2Y5MzUtNjI4YjEzZGI=, ActorId: [3:7569244409802246590:2580], ActorState: ExecuteState, TraceId: 01k9a5w9ce9q8vhbht8m1dxs2e, Create QueryResponse for error on request, msg: 
2025-11-05T14:14:08.017956Z node 3 :PERSQUEUE_CLUSTER_TRACKER ERROR: failed to list clusters: { Response { QueryIssues { message: "Execution" issue_code: 1060 severity: 2 issues { position { row: 3 column: 120 } message: "Cost Based Optimizer could not be applied to this query: couldn\'t load statistics" end_position { row: 3 column: 120 } issue_code: 8001 severity: 2 } } QueryIssues { message: "Failed to resolve tablet: 72075186224037890 after several retries." severity: 1 } TxMeta { id: "01k9a5w9pw5n1a4gjhjdnq6rrt" } } YdbStatus: UNAVAILABLE ConsumedRu: 219 }
2025-11-05T14:14:08.116820Z node 3 :KQP_COMPUTE WARN: SelfId: [3:7569244409802246623:2586], TxId: 281474976715704, task: 1, CA Id [3:7569244409802246621:2586]. Got EvDeliveryProblem, TabletId: 72075186224037890, NotDelivered: 1
2025-11-05T14:14:08.152063Z node 3 :FLAT_TX_SCHEMESHARD WARN: Cannot get console configs
2025-11-05T14:14:08.152091Z node 3 :IMPORT WARN: Table profiles were not loaded
2025-11-05T14:14:08.364812Z node 3 :KQP_COMPUTE WARN: SelfId: [3:7569244409802246623:2586], TxId: 281474976715704, task: 1, CA Id [3:7569244409802246621:2586]. Got EvDeliveryProblem, TabletId: 72075186224037890, NotDelivered: 1
