New paste Repaste Download
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.10.14, pytest-8.3.5, pluggy-1.5.0 -- /home/rjagarla/work/pytest-automation/new_runner/bin/python
rootdir: /home/rjagarla/work/pytest-automation/mdt-automation
configfile: pytest.ini
plugins: jira-0.3.21, mdt-2.1.5.dev21+gf5e1951, alsi-3.1.4
collecting ...
collected 6 items                                                                                                                                                                                                 
tests/mdtreceiver/test_channel_limits_new.py::test_default_limits_are_set_in_mdtreceiver_config
------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------
2025-06-27 07:21:43 DEBUG root.mdt_network_inventory_fixture:119 - Adding 'metanet' to inventory.
2025-06-27 07:21:43 INFO python_alsi.testnet.networks:272 - Loading networks from testnet sqa2.
2025-06-27 07:21:43 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=<python_alsi.testnet.DirectTestnet object at 0x7f8d9685a710>,name=freeflow,fetcher=<python_alsi.perforce.PathFetcher object at 0x7f8d9685a860>
2025-06-27 07:21:54 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=<python_alsi.testnet.DirectTestnet object at 0x7f8d9685a710>,name=infra,fetcher=<python_alsi.perforce.PathFetcher object at 0x7f8d9685a860>
2025-06-27 07:21:57 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=<python_alsi.testnet.DirectTestnet object at 0x7f8d9685a710>,name=ff,fetcher=<python_alsi.perforce.PathFetcher object at 0x7f8d9685a860>
2025-06-27 07:22:01 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=<python_alsi.testnet.DirectTestnet object at 0x7f8d9685a710>,name=ness,fetcher=<python_alsi.perforce.PathFetcher object at 0x7f8d9685a860>
2025-06-27 07:22:05 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=<python_alsi.testnet.DirectTestnet object at 0x7f8d9685a710>,name=essl,fetcher=<python_alsi.perforce.PathFetcher object at 0x7f8d9685a860>
2025-06-27 07:22:08 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=<python_alsi.testnet.DirectTestnet object at 0x7f8d9685a710>,name=volta,fetcher=<python_alsi.perforce.PathFetcher object at 0x7f8d9685a860>
2025-06-27 07:22:12 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=<python_alsi.testnet.DirectTestnet object at 0x7f8d9685a710>,name=metanet,fetcher=<python_alsi.perforce.PathFetcher object at 0x7f8d9685a860>
2025-06-27 07:22:15 INFO python_alsi.testnet.networks:277 - Loaded 7 networks from testnet sqa2.
2025-06-27 07:22:15 DEBUG python_mdt.system.mdtsystem._system_info_setup:178 - Setting up MDT system: 'freeflow' in Network: 'metanet'
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.8.7
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.8.7,nat_gateway=None,authgate=None
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.3.90
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.3.90,nat_gateway=None,authgate=None
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.11.120
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.11.120,nat_gateway=None,authgate=None
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.16.14
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.16.14,nat_gateway=None,authgate=None
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.101.101
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.101.101,nat_gateway=None,authgate=None
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.101.102
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.101.102,nat_gateway=None,authgate=None
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.171
2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.171,nat_gateway=None,authgate=None
2025-06-27 07:22:16 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.171] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:22:16 INFO python_alsi.connection.connection:168 - [198.18.116.171] Connecting as user root
2025-06-27 07:22:16 DEBUG python_alsi.connection.connection:193 - [198.18.116.171] Connecting directly using ssh
2025-06-27 07:22:16 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.171']
2025-06-27 07:22:19 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.171
2025-06-27 07:22:20 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:22:20 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:22:21 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.171', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:22:24 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:22:26 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.171] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:22:28 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:22:28 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:22:28 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.174
2025-06-27 07:22:28 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.174,nat_gateway=None,authgate=None
2025-06-27 07:22:28 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.174] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:22:28 INFO python_alsi.connection.connection:168 - [198.18.116.174] Connecting as user root
2025-06-27 07:22:28 DEBUG python_alsi.connection.connection:193 - [198.18.116.174] Connecting directly using ssh
2025-06-27 07:22:28 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.174']
2025-06-27 07:22:32 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.174
2025-06-27 07:22:32 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:22:32 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:22:33 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.174', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:22:36 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:22:38 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.174] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:22:40 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:22:40 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:22:40 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.177
2025-06-27 07:22:40 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.177,nat_gateway=None,authgate=None
2025-06-27 07:22:40 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.177] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:22:40 INFO python_alsi.connection.connection:168 - [198.18.116.177] Connecting as user root
2025-06-27 07:22:40 DEBUG python_alsi.connection.connection:193 - [198.18.116.177] Connecting directly using ssh
2025-06-27 07:22:40 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.177']
2025-06-27 07:22:44 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.177
2025-06-27 07:22:44 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:22:44 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:22:45 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.177', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:22:48 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:22:50 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.177] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:22:52 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_3_198.18.116.177.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_3_198.18.116.177.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_3_198.18.116.177.json')}
2025-06-27 07:22:52 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_3_198.18.116.177.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_3_198.18.116.177.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_3_198.18.116.177.json')}
2025-06-27 07:22:52 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.228.115
2025-06-27 07:22:52 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.228.115,nat_gateway=None,authgate=None
2025-06-27 07:22:52 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.115] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:22:52 INFO python_alsi.connection.connection:168 - [198.18.228.115] Connecting as user root
2025-06-27 07:22:52 DEBUG python_alsi.connection.connection:193 - [198.18.228.115] Connecting directly using ssh
2025-06-27 07:22:52 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.115']
2025-06-27 07:22:56 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.228.115
2025-06-27 07:22:56 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:22:56 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:22:58 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.115', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:23:00 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:23:03 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.115] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:23:04 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_4_198.18.228.115.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_4_198.18.228.115.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_4_198.18.228.115.json')}
2025-06-27 07:23:04 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_4_198.18.228.115.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_4_198.18.228.115.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_4_198.18.228.115.json')}
2025-06-27 07:23:04 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.11.121
2025-06-27 07:23:04 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.11.121,nat_gateway=None,authgate=None
2025-06-27 07:23:04 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.121] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:23:04 INFO python_alsi.connection.connection:168 - [198.19.11.121] Connecting as user root
2025-06-27 07:23:04 DEBUG python_alsi.connection.connection:193 - [198.19.11.121] Connecting directly using ssh
2025-06-27 07:23:04 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.121']
2025-06-27 07:23:08 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.11.121
2025-06-27 07:23:09 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:23:09 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:23:10 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.121', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:23:12 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:23:15 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.121] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:23:16 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:23:16 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:23:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.172
2025-06-27 07:23:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.172,nat_gateway=None,authgate=None
2025-06-27 07:23:16 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.172] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:23:16 INFO python_alsi.connection.connection:168 - [198.18.116.172] Connecting as user root
2025-06-27 07:23:16 DEBUG python_alsi.connection.connection:193 - [198.18.116.172] Connecting directly using ssh
2025-06-27 07:23:16 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.172']
2025-06-27 07:23:20 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.172
2025-06-27 07:23:21 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:23:21 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:23:22 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.172', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:23:25 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:23:27 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.172] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:23:29 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:23:29 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:23:29 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.173
2025-06-27 07:23:29 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.173,nat_gateway=None,authgate=None
2025-06-27 07:23:29 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.173] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:23:29 INFO python_alsi.connection.connection:168 - [198.18.116.173] Connecting as user root
2025-06-27 07:23:29 DEBUG python_alsi.connection.connection:193 - [198.18.116.173] Connecting directly using ssh
2025-06-27 07:23:29 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.173']
2025-06-27 07:23:32 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.173
2025-06-27 07:23:33 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:23:33 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:23:34 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.173', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:23:37 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:23:39 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.173] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:23:41 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:23:41 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:23:41 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtOriginMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.178
2025-06-27 07:23:41 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.178,nat_gateway=None,authgate=None
2025-06-27 07:23:41 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.178] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:23:41 INFO python_alsi.connection.connection:168 - [198.18.116.178] Connecting as user root
2025-06-27 07:23:41 DEBUG python_alsi.connection.connection:193 - [198.18.116.178] Connecting directly using ssh
2025-06-27 07:23:41 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.178']
2025-06-27 07:23:44 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.178
2025-06-27 07:23:45 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:23:45 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:23:46 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.178', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:23:49 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:23:51 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.178] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:23:53 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None}
2025-06-27 07:23:53 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None}
2025-06-27 07:23:53 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.175
2025-06-27 07:23:53 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.175,nat_gateway=None,authgate=None
2025-06-27 07:23:53 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.175] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:23:53 INFO python_alsi.connection.connection:168 - [198.18.116.175] Connecting as user root
2025-06-27 07:23:53 DEBUG python_alsi.connection.connection:193 - [198.18.116.175] Connecting directly using ssh
2025-06-27 07:23:53 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.175']
2025-06-27 07:23:57 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.175
2025-06-27 07:23:57 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:23:57 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:23:58 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.175', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:24:01 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:24:03 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.175] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:24:05 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_1_198.18.116.175.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_1_198.18.116.175.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_1_198.18.116.175.json')}
2025-06-27 07:24:05 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_1_198.18.116.175.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_1_198.18.116.175.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_1_198.18.116.175.json')}
2025-06-27 07:24:05 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.176
2025-06-27 07:24:05 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.176,nat_gateway=None,authgate=None
2025-06-27 07:24:05 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.176] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:24:05 INFO python_alsi.connection.connection:168 - [198.18.116.176] Connecting as user root
2025-06-27 07:24:05 DEBUG python_alsi.connection.connection:193 - [198.18.116.176] Connecting directly using ssh
2025-06-27 07:24:05 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.176']
2025-06-27 07:24:09 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.176
2025-06-27 07:24:09 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:24:09 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:24:10 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.176', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:24:13 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:24:15 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.176] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:24:17 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_2_198.18.116.176.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_2_198.18.116.176.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_2_198.18.116.176.json')}
2025-06-27 07:24:17 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_2_198.18.116.176.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_2_198.18.116.176.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_2_198.18.116.176.json')}
2025-06-27 07:24:17 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.167.214
2025-06-27 07:24:17 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.167.214,nat_gateway=None,authgate=None
2025-06-27 07:24:17 DEBUG python_alsi.machine.run_cmd:270 - [198.18.167.214] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:24:17 INFO python_alsi.connection.connection:168 - [198.18.167.214] Connecting as user root
2025-06-27 07:24:17 DEBUG python_alsi.connection.connection:193 - [198.18.167.214] Connecting directly using ssh
2025-06-27 07:24:17 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.167.214']
2025-06-27 07:24:21 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.167.214
2025-06-27 07:24:21 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:24:21 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:24:23 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.167.214', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:24:25 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:24:28 DEBUG python_alsi.machine.run_cmd:270 - [198.18.167.214] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:24:29 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:24:29 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:24:29 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.168
2025-06-27 07:24:29 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.168,nat_gateway=None,authgate=None
2025-06-27 07:24:29 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:24:29 INFO python_alsi.connection.connection:168 - [198.18.116.168] Connecting as user root
2025-06-27 07:24:29 DEBUG python_alsi.connection.connection:193 - [198.18.116.168] Connecting directly using ssh
2025-06-27 07:24:29 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.168']
2025-06-27 07:24:33 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.168
2025-06-27 07:24:34 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:24:34 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:24:35 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.168', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:24:37 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:24:40 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:24:41 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:24:41 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:24:41 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtOriginMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.179
2025-06-27 07:24:41 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.179,nat_gateway=None,authgate=None
2025-06-27 07:24:41 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.179] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:24:41 INFO python_alsi.connection.connection:168 - [198.18.116.179] Connecting as user root
2025-06-27 07:24:41 DEBUG python_alsi.connection.connection:193 - [198.18.116.179] Connecting directly using ssh
2025-06-27 07:24:41 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.179']
2025-06-27 07:24:45 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.179
2025-06-27 07:24:46 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:24:46 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:24:47 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.179', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:24:50 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:24:52 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.179] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:24:53 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None}
2025-06-27 07:24:53 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None}
2025-06-27 07:24:53 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.228.116
2025-06-27 07:24:53 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.228.116,nat_gateway=None,authgate=None
2025-06-27 07:24:53 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.116] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:24:53 INFO python_alsi.connection.connection:168 - [198.18.228.116] Connecting as user root
2025-06-27 07:24:53 DEBUG python_alsi.connection.connection:193 - [198.18.228.116] Connecting directly using ssh
2025-06-27 07:24:53 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.116']
2025-06-27 07:24:57 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.228.116
2025-06-27 07:24:58 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:24:58 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:24:59 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.116', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:25:02 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:25:04 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.116] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:25:06 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_5_198.18.228.116.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_5_198.18.228.116.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_5_198.18.228.116.json')}
2025-06-27 07:25:06 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_5_198.18.228.116.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_5_198.18.228.116.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_5_198.18.228.116.json')}
2025-06-27 07:25:06 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.11.122
2025-06-27 07:25:06 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.11.122,nat_gateway=None,authgate=None
2025-06-27 07:25:06 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.122] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:25:06 INFO python_alsi.connection.connection:168 - [198.19.11.122] Connecting as user root
2025-06-27 07:25:06 DEBUG python_alsi.connection.connection:193 - [198.19.11.122] Connecting directly using ssh
2025-06-27 07:25:06 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.122']
2025-06-27 07:25:09 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.11.122
2025-06-27 07:25:10 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:25:10 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:25:11 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.122', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:25:14 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:25:16 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.122] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:25:18 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:25:18 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:25:18 INFO python_mdt.system.mdtsystem._inventory_machines:245 - Ignoring machine 198.18.116.169 from inventory
2025-06-27 07:25:18 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.228.117
2025-06-27 07:25:18 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.228.117,nat_gateway=None,authgate=None
2025-06-27 07:25:18 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.117] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:25:18 INFO python_alsi.connection.connection:168 - [198.18.228.117] Connecting as user root
2025-06-27 07:25:18 DEBUG python_alsi.connection.connection:193 - [198.18.228.117] Connecting directly using ssh
2025-06-27 07:25:18 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.117']
2025-06-27 07:25:22 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.228.117
2025-06-27 07:25:22 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:25:22 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:25:23 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.117', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:25:26 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:25:28 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.117] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:25:30 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_6_198.18.228.117.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_6_198.18.228.117.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_6_198.18.228.117.json')}
2025-06-27 07:25:30 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_6_198.18.228.117.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_6_198.18.228.117.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_6_198.18.228.117.json')}
2025-06-27 07:25:30 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.41.7
2025-06-27 07:25:30 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.41.7,nat_gateway=None,authgate=None
2025-06-27 07:25:30 DEBUG python_alsi.machine.run_cmd:270 - [198.18.41.7] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:25:30 INFO python_alsi.connection.connection:168 - [198.18.41.7] Connecting as user root
2025-06-27 07:25:30 DEBUG python_alsi.connection.connection:193 - [198.18.41.7] Connecting directly using ssh
2025-06-27 07:25:30 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.41.7']
2025-06-27 07:25:34 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.41.7
2025-06-27 07:25:34 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:25:34 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:25:36 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.41.7', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:25:38 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:25:41 DEBUG python_alsi.machine.run_cmd:270 - [198.18.41.7] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:25:42 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:25:42 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:25:42 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.116.170
2025-06-27 07:25:42 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.170,nat_gateway=None,authgate=None
2025-06-27 07:25:42 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.170] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:25:42 INFO python_alsi.connection.connection:168 - [198.18.116.170] Connecting as user root
2025-06-27 07:25:42 DEBUG python_alsi.connection.connection:193 - [198.18.116.170] Connecting directly using ssh
2025-06-27 07:25:42 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.170']
2025-06-27 07:25:46 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.170
2025-06-27 07:25:47 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:25:47 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:25:48 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.170', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:25:50 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:25:53 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.170] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:25:54 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:25:54 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:25:54 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.228.120
2025-06-27 07:25:54 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.228.120,nat_gateway=None,authgate=None
2025-06-27 07:25:54 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.120] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:25:54 INFO python_alsi.connection.connection:168 - [198.18.228.120] Connecting as user root
2025-06-27 07:25:54 DEBUG python_alsi.connection.connection:193 - [198.18.228.120] Connecting directly using ssh
2025-06-27 07:25:54 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.120']
2025-06-27 07:25:58 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.228.120
2025-06-27 07:25:59 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:25:59 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:26:00 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.120', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:26:03 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:26:05 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.120] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:26:07 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_7_198.18.228.120.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_7_198.18.228.120.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_7_198.18.228.120.json')}
2025-06-27 07:26:07 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_7_198.18.228.120.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_7_198.18.228.120.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_7_198.18.228.120.json')}
2025-06-27 07:26:07 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.11.123
2025-06-27 07:26:07 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.11.123,nat_gateway=None,authgate=None
2025-06-27 07:26:07 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.123] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:26:07 INFO python_alsi.connection.connection:168 - [198.19.11.123] Connecting as user root
2025-06-27 07:26:07 DEBUG python_alsi.connection.connection:193 - [198.19.11.123] Connecting directly using ssh
2025-06-27 07:26:07 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.123']
2025-06-27 07:26:10 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.11.123
2025-06-27 07:26:11 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:26:11 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:26:12 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.123', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:26:15 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:26:17 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.123] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:26:19 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:26:19 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:26:19 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.16.114
2025-06-27 07:26:19 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.16.114,nat_gateway=None,authgate=None
2025-06-27 07:26:19 DEBUG python_alsi.machine.run_cmd:270 - [198.19.16.114] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:26:19 INFO python_alsi.connection.connection:168 - [198.19.16.114] Connecting as user root
2025-06-27 07:26:19 DEBUG python_alsi.connection.connection:193 - [198.19.16.114] Connecting directly using ssh
2025-06-27 07:26:19 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.16.114']
2025-06-27 07:26:23 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.16.114
2025-06-27 07:26:23 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:26:23 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:26:24 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.16.114', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:26:27 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:26:29 DEBUG python_alsi.machine.run_cmd:270 - [198.19.16.114] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:26:31 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:26:31 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:26:31 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.16.116
2025-06-27 07:26:31 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.16.116,nat_gateway=None,authgate=None
2025-06-27 07:26:31 DEBUG python_alsi.machine.run_cmd:270 - [198.19.16.116] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:26:31 INFO python_alsi.connection.connection:168 - [198.19.16.116] Connecting as user root
2025-06-27 07:26:31 DEBUG python_alsi.connection.connection:193 - [198.19.16.116] Connecting directly using ssh
2025-06-27 07:26:31 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.16.116']
2025-06-27 07:26:35 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.16.116
2025-06-27 07:26:35 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:26:35 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:26:37 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.16.116', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:26:40 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:26:42 DEBUG python_alsi.machine.run_cmd:270 - [198.19.16.116] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:26:44 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:26:44 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:26:44 INFO root.mdt_system_inventory_fixture:142 - As requested, 'mdt-test-auto' won't be included in the inventory.
2025-06-27 07:26:44 INFO root.mdt_system_inventory_fixture:142 - As requested, 'mdt-test2' won't be included in the inventory.
2025-06-27 07:26:44 DEBUG python_mdt.system.mdtsystem._system_info_setup:178 - Setting up MDT system: 'mdt-test3' in Network: 'metanet'
2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.8.7
2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.3.90
2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.11.120
2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.19.16.14
2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.88.154
2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.88.154,nat_gateway=None,authgate=None
2025-06-27 07:26:45 DEBUG python_alsi.machine.run_cmd:270 - [198.18.88.154] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:26:45 INFO python_alsi.connection.connection:168 - [198.18.88.154] Connecting as user root
2025-06-27 07:26:45 DEBUG python_alsi.connection.connection:193 - [198.18.88.154] Connecting directly using ssh
2025-06-27 07:26:45 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.88.154']
2025-06-27 07:26:49 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.88.154
2025-06-27 07:26:50 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:26:50 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:26:51 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.88.154', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:26:54 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:26:56 DEBUG python_alsi.machine.run_cmd:270 - [198.18.88.154] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:26:58 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:26:58 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:26:58 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.93.176
2025-06-27 07:26:58 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.93.176,nat_gateway=None,authgate=None
2025-06-27 07:26:58 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:26:58 INFO python_alsi.connection.connection:168 - [198.18.93.176] Connecting as user root
2025-06-27 07:26:58 DEBUG python_alsi.connection.connection:193 - [198.18.93.176] Connecting directly using ssh
2025-06-27 07:26:58 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.93.176']
2025-06-27 07:27:01 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.93.176
2025-06-27 07:27:02 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:27:02 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:27:03 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.93.176', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:27:06 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:27:08 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:27:10 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:27:10 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:27:10 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.101.99
2025-06-27 07:27:10 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.101.99,nat_gateway=None,authgate=None
2025-06-27 07:27:10 DEBUG python_alsi.machine.run_cmd:270 - [198.18.101.99] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:27:10 INFO python_alsi.connection.connection:168 - [198.18.101.99] Connecting as user root
2025-06-27 07:27:10 DEBUG python_alsi.connection.connection:193 - [198.18.101.99] Connecting directly using ssh
2025-06-27 07:27:10 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.101.99']
2025-06-27 07:27:14 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.101.99
2025-06-27 07:27:14 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:27:14 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:27:15 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.101.99', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:27:18 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:27:20 DEBUG python_alsi.machine.run_cmd:270 - [198.18.101.99] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:27:22 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:27:22 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': None}
2025-06-27 07:27:22 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.102.196
2025-06-27 07:27:22 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.102.196,nat_gateway=None,authgate=None
2025-06-27 07:27:22 DEBUG python_alsi.machine.run_cmd:270 - [198.18.102.196] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:27:22 INFO python_alsi.connection.connection:168 - [198.18.102.196] Connecting as user root
2025-06-27 07:27:22 DEBUG python_alsi.connection.connection:193 - [198.18.102.196] Connecting directly using ssh
2025-06-27 07:27:22 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.102.196']
2025-06-27 07:27:26 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.102.196
2025-06-27 07:27:26 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:27:26 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:27:28 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.102.196', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:27:30 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:27:33 DEBUG python_alsi.machine.run_cmd:270 - [198.18.102.196] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:27:34 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_1_198.18.102.196.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_1_198.18.102.196.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_1_198.18.102.196.json')}
2025-06-27 07:27:34 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_1_198.18.102.196.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_1_198.18.102.196.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_1_198.18.102.196.json')}
2025-06-27 07:27:34 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.55.186
2025-06-27 07:27:34 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.55.186,nat_gateway=None,authgate=None
2025-06-27 07:27:34 DEBUG python_alsi.machine.run_cmd:270 - [198.18.55.186] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:27:34 INFO python_alsi.connection.connection:168 - [198.18.55.186] Connecting as user root
2025-06-27 07:27:34 DEBUG python_alsi.connection.connection:193 - [198.18.55.186] Connecting directly using ssh
2025-06-27 07:27:34 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.55.186']
2025-06-27 07:27:38 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.55.186
2025-06-27 07:27:38 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:27:38 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:27:40 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.55.186', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:27:42 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:27:45 DEBUG python_alsi.machine.run_cmd:270 - [198.18.55.186] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:27:46 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_2_198.18.55.186.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_2_198.18.55.186.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_2_198.18.55.186.json')}
2025-06-27 07:27:46 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_2_198.18.55.186.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_2_198.18.55.186.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_2_198.18.55.186.json')}
2025-06-27 07:27:46 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.88.252
2025-06-27 07:27:46 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.88.252,nat_gateway=None,authgate=None
2025-06-27 07:27:46 DEBUG python_alsi.machine.run_cmd:270 - [198.18.88.252] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:27:46 INFO python_alsi.connection.connection:168 - [198.18.88.252] Connecting as user root
2025-06-27 07:27:46 DEBUG python_alsi.connection.connection:193 - [198.18.88.252] Connecting directly using ssh
2025-06-27 07:27:46 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.88.252']
2025-06-27 07:27:50 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.88.252
2025-06-27 07:27:51 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:27:51 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:27:52 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.88.252', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:27:54 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:27:57 DEBUG python_alsi.machine.run_cmd:270 - [198.18.88.252] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:27:58 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_3_198.18.88.252.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_3_198.18.88.252.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_3_198.18.88.252.json')}
2025-06-27 07:27:58 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_3_198.18.88.252.json'),
  'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_3_198.18.88.252.json'),
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'),
  'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_3_198.18.88.252.json')}
2025-06-27 07:27:58 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.180.138
2025-06-27 07:27:58 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.180.138,nat_gateway=None,authgate=None
2025-06-27 07:27:58 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.138] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:27:58 INFO python_alsi.connection.connection:168 - [198.18.180.138] Connecting as user root
2025-06-27 07:27:58 DEBUG python_alsi.connection.connection:193 - [198.18.180.138] Connecting directly using ssh
2025-06-27 07:27:58 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.138']
2025-06-27 07:28:02 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.180.138
2025-06-27 07:28:03 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:28:03 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:28:04 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.138', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:28:06 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:28:09 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.138] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:28:10 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:28:10 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:28:10 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.180.141
2025-06-27 07:28:10 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.180.141,nat_gateway=None,authgate=None
2025-06-27 07:28:10 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.141] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:28:10 INFO python_alsi.connection.connection:168 - [198.18.180.141] Connecting as user root
2025-06-27 07:28:10 DEBUG python_alsi.connection.connection:193 - [198.18.180.141] Connecting directly using ssh
2025-06-27 07:28:10 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.141']
2025-06-27 07:28:14 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.180.141
2025-06-27 07:28:15 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:28:15 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:28:16 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.141', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:28:19 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:28:21 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.141] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:28:23 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:28:23 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:28:23 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.93.197
2025-06-27 07:28:23 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.93.197,nat_gateway=None,authgate=None
2025-06-27 07:28:23 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.197] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:28:23 INFO python_alsi.connection.connection:168 - [198.18.93.197] Connecting as user root
2025-06-27 07:28:23 DEBUG python_alsi.connection.connection:193 - [198.18.93.197] Connecting directly using ssh
2025-06-27 07:28:23 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.93.197']
2025-06-27 07:28:26 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.93.197
2025-06-27 07:28:27 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:28:27 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:28:28 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.93.197', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:28:31 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:28:33 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.197] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:28:35 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:28:35 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:28:35 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.180.134
2025-06-27 07:28:35 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.180.134,nat_gateway=None,authgate=None
2025-06-27 07:28:35 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.134] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:28:35 INFO python_alsi.connection.connection:168 - [198.18.180.134] Connecting as user root
2025-06-27 07:28:35 DEBUG python_alsi.connection.connection:193 - [198.18.180.134] Connecting directly using ssh
2025-06-27 07:28:35 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.134']
2025-06-27 07:28:38 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.180.134
2025-06-27 07:28:39 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:28:39 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:28:40 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.134', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:28:43 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:28:45 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.134] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:28:47 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{ 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:28:47 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{ 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'),
  'debug': None,
  'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'),
  'quorum': None}
2025-06-27 07:28:47 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtOriginMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.101.100
2025-06-27 07:28:47 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.101.100,nat_gateway=None,authgate=None
2025-06-27 07:28:47 DEBUG python_alsi.machine.run_cmd:270 - [198.18.101.100] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root
2025-06-27 07:28:47 INFO python_alsi.connection.connection:168 - [198.18.101.100] Connecting as user root
2025-06-27 07:28:47 DEBUG python_alsi.connection.connection:193 - [198.18.101.100] Connecting directly using ssh
2025-06-27 07:28:47 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.101.100']
2025-06-27 07:28:51 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.101.100
2025-06-27 07:28:51 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell.
2025-06-27 07:28:51 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11).
2025-06-27 07:28:52 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.101.100', 'sh', '-c', '"echo SYNC0;exec sh"']
2025-06-27 07:28:55 DEBUG python_alsi.connection.__init__:696 - Connected.
2025-06-27 07:28:57 DEBUG python_alsi.machine.run_cmd:270 - [198.18.101.100] Running 'ls /a/ssl-rotation-manager/state/*.json' as root
2025-06-27 07:28:59 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files
{'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None}
2025-06-27 07:28:59 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files
{'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None}
2025-06-27 07:28:59 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.101.101
2025-06-27 07:28:59 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=<python_alsi.network.DirectNetwork object at 0x7f8d9622fcd0>,ip=198.18.101.102
2025-06-27 07:28:59 INFO python_mdt.system.mdtsystem._inventory_machines:245 - Ignoring machine 198.18.116.169 from inventory
2025-06-27 07:28:59 INFO root.mdt_channel_injector_fixture:489 - Injecting MDT channels into the MDT system.
2025-06-27 07:28:59 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtZooKeeperHelper -  ['dynamic_quorum_config_data', 'servers_post_subscription_config_data', 'servers_pre_subscription_config_data']
2025-06-27 07:28:59 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.116.175, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:29:04 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtzookeeper.quorum.freeflow' MDT system.
2025-06-27 07:29:04 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.168, network=metanet, testnet=sqa2, role=server
2025-06-27 07:29:10 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtzookeeper.quorum.freeflow'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:29:15 DEBUG python_alsi.service.get_local_tableprov_data:414 - Found matching row for machine 198.18.116.168 in table mdtreceiver_status: {'machineip': '198.18.116.168', 'version': '2.22', 'reporting_time': '1751009322', 'client_connection_limit': '110', 'client_connections': '34', 'max_client_connections_1h': '35', 'heap_used': '338684256', 'heap_committed': '1090519040', 'heap_max': '12884901888', 'nonheap_used': '105102728', 'nonheap_committed': '138674176', 'bad_zk_state_1h': '0', 'mdtzookeeper_quorum': 'mdtzookeeper.sqa2.qa.akamai.com', 'zk_op_successes_1m': '9', 'zk_op_failures_1m': '0', 'zk_op_errors_1m': '0', 'zk_op_successes_5m': '122', 'zk_op_failures_5m': '0', 'zk_op_errors_5m': '0', 'zk_op_successes_15m': '320', 'zk_op_failures_15m': '0', 'zk_op_errors_15m': '0'}
2025-06-27 07:29:16 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'source /usr/local/akamai/etc/akamai.conf; $MDT_JAVA_HOME/bin/keytool -importkeystore -srckeystore /usr/local/akamai/mdtreceiver/etc/keys/api.jks -srcstorepass password -destkeystore /tmp/python-mdt-secrets/api.p12.tmp -deststorepass password -deststoretype pkcs12 -noprompt -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /a/mdtreceiver/lib/bc-fips.jar -srcstoretype BCFKS' as root
2025-06-27 07:29:19 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'openssl pkcs12 -in /tmp/python-mdt-secrets/api.p12 -nodes -nocerts -out /tmp/python-mdt-secrets/api.private_key.tmp -passin pass:password' as root
2025-06-27 07:29:20 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'openssl pkcs12 -in /tmp/python-mdt-secrets/api.p12 -nodes -nokeys -out /tmp/python-mdt-secrets/api.certificate.tmp -passin pass:password' as root
2025-06-27 07:29:23 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtzookeeper_servers.postinstall.csv' MDT system.
2025-06-27 07:29:24 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:29:24 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtzookeeper_servers.preinstall.csv' MDT system.
2025-06-27 07:29:26 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:29:26 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'dynamic_quorum_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3d660>, 'servers_post_subscription_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3e1d0>, 'servers_pre_subscription_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3ef50>}
2025-06-27 07:29:26 INFO root.inject_channels:466 - MDTZK ip: 198.18.116.175
2025-06-27 07:29:26 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3d660>)
2025-06-27 07:29:26 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3e1d0>)
2025-06-27 07:29:26 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3ef50>)
2025-06-27 07:29:26 INFO root.inject_channels:466 - MDTZK ip: 198.18.228.117
2025-06-27 07:29:26 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.228.117, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:29:31 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3d660>)
2025-06-27 07:29:31 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3e1d0>)
2025-06-27 07:29:31 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3ef50>)
2025-06-27 07:29:31 INFO root.inject_channels:466 - MDTZK ip: 198.18.116.177
2025-06-27 07:29:31 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.116.177, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:29:36 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3d660>)
2025-06-27 07:29:36 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3e1d0>)
2025-06-27 07:29:36 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3ef50>)
2025-06-27 07:29:36 INFO root.inject_channels:466 - MDTZK ip: 198.18.228.115
2025-06-27 07:29:36 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.228.115, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:29:41 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3d660>)
2025-06-27 07:29:41 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3e1d0>)
2025-06-27 07:29:41 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3ef50>)
2025-06-27 07:29:41 INFO root.inject_channels:466 - MDTZK ip: 198.18.228.120
2025-06-27 07:29:41 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.228.120, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:29:46 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3d660>)
2025-06-27 07:29:46 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3e1d0>)
2025-06-27 07:29:46 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3ef50>)
2025-06-27 07:29:46 INFO root.inject_channels:466 - MDTZK ip: 198.18.228.116
2025-06-27 07:29:46 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.228.116, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:29:51 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3d660>)
2025-06-27 07:29:51 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3e1d0>)
2025-06-27 07:29:51 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3ef50>)
2025-06-27 07:29:51 INFO root.inject_channels:466 - MDTZK ip: 198.18.116.176
2025-06-27 07:29:51 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.116.176, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:29:56 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3d660>)
2025-06-27 07:29:56 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3e1d0>)
2025-06-27 07:29:56 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95e3ef50>)
2025-06-27 07:29:56 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtProducer -  ['ensemble_channel_config_data', 'servers_post_channel_config_data', 'servers_pre_channel_config_data']
2025-06-27 07:29:56 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.11.122, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:30:01 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtproducer.sets.producer_ensemble1' MDT system.
2025-06-27 07:30:02 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtproducer.sets.producer_ensemble1'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:30:02 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtproducer_servers.postinstall.csv' MDT system.
2025-06-27 07:30:04 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtproducer_servers.postinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:30:04 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtproducer_servers.preinstall.csv' MDT system.
2025-06-27 07:30:05 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtproducer_servers.preinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:30:05 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'ensemble_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95be6650>, 'servers_post_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95be6a40>, 'servers_pre_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95be6c80>}
2025-06-27 07:30:05 INFO root.inject_channels:476 - MDTProducer ip: 198.19.11.122
2025-06-27 07:30:05 INFO root.inject_channels:476 - MDTProducer ip: 198.18.167.214
2025-06-27 07:30:05 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.167.214, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:30:10 INFO root.inject_channels:476 - MDTProducer ip: 198.18.41.7
2025-06-27 07:30:10 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.41.7, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:30:15 INFO root.inject_channels:476 - MDTProducer ip: 198.19.11.121
2025-06-27 07:30:15 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.11.121, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:30:20 INFO root.inject_channels:476 - MDTProducer ip: 198.19.11.123
2025-06-27 07:30:20 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.11.123, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:30:25 INFO root.inject_channels:476 - MDTProducer ip: 198.19.16.116
2025-06-27 07:30:25 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.16.116, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:30:30 INFO root.inject_channels:476 - MDTProducer ip: 198.19.16.114
2025-06-27 07:30:30 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.16.114, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:30:35 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtReceiver -  ['dynamic_channel_config_data', 'group_channel_config_data']
2025-06-27 07:30:35 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtreceiver.freeflow' MDT system.
2025-06-27 07:30:36 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtreceiver.freeflow'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:30:36 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.groups.freeflow' MDT system.
2025-06-27 07:30:38 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.groups.freeflow'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:30:38 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'dynamic_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95a6beb0>, 'group_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95a8d8d0>}
2025-06-27 07:30:38 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.168
2025-06-27 07:30:38 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.173
2025-06-27 07:30:38 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.173, network=metanet, testnet=sqa2, role=server
2025-06-27 07:30:43 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.171
2025-06-27 07:30:43 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.171, network=metanet, testnet=sqa2, role=server
2025-06-27 07:30:48 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.172
2025-06-27 07:30:48 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.172, network=metanet, testnet=sqa2, role=server
2025-06-27 07:30:53 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.170
2025-06-27 07:30:53 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.170, network=metanet, testnet=sqa2, role=server
2025-06-27 07:30:58 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.174
2025-06-27 07:30:58 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.174, network=metanet, testnet=sqa2, role=server
2025-06-27 07:31:03 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtZooKeeperHelper -  ['dynamic_quorum_config_data', 'servers_post_subscription_config_data', 'servers_pre_subscription_config_data']
2025-06-27 07:31:03 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.55.186, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:31:08 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtzookeeper.quorum.mdt-test3' MDT system.
2025-06-27 07:31:09 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtzookeeper.quorum.mdt-test3'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:31:09 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtzookeeper_servers.postinstall.csv' MDT system.
2025-06-27 07:31:11 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:31:11 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtzookeeper_servers.preinstall.csv' MDT system.
2025-06-27 07:31:12 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:31:12 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'dynamic_quorum_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95954100>, 'servers_post_subscription_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95955bd0>, 'servers_pre_subscription_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95955ea0>}
2025-06-27 07:31:12 INFO root.inject_channels:466 - MDTZK ip: 198.18.55.186
2025-06-27 07:31:12 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.mdt-test3', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95954100>)
2025-06-27 07:31:12 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95955bd0>)
2025-06-27 07:31:12 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95955ea0>)
2025-06-27 07:31:12 INFO root.inject_channels:466 - MDTZK ip: 198.18.88.252
2025-06-27 07:31:12 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.88.252, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:31:17 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.mdt-test3', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95954100>)
2025-06-27 07:31:17 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95955bd0>)
2025-06-27 07:31:17 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95955ea0>)
2025-06-27 07:31:17 INFO root.inject_channels:466 - MDTZK ip: 198.18.102.196
2025-06-27 07:31:17 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.102.196, network=metanet, testnet=sqa2, role=zookeeper
2025-06-27 07:31:22 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.mdt-test3', port=5874, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95954100>)
2025-06-27 07:31:22 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95955bd0>)
2025-06-27 07:31:22 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=<python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95955ea0>)
2025-06-27 07:31:22 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtProducer -  ['ensemble_channel_config_data', 'servers_post_channel_config_data', 'servers_pre_channel_config_data']
2025-06-27 07:31:22 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.180.141, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:31:27 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtproducer.sets.producer_ensemble_test3' MDT system.
2025-06-27 07:31:29 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtproducer.sets.producer_ensemble_test3'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:31:29 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtproducer_servers.postinstall.csv' MDT system.
2025-06-27 07:31:30 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtproducer_servers.postinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:31:30 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtproducer_servers.preinstall.csv' MDT system.
2025-06-27 07:31:31 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtproducer_servers.preinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:31:31 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'ensemble_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d95977ee0>, 'servers_post_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d959859c0>, 'servers_pre_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d9590feb0>}
2025-06-27 07:31:31 INFO root.inject_channels:476 - MDTProducer ip: 198.18.180.141
2025-06-27 07:31:31 INFO root.inject_channels:476 - MDTProducer ip: 198.18.180.138
2025-06-27 07:31:31 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.180.138, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:31:36 INFO root.inject_channels:476 - MDTProducer ip: 198.18.180.134
2025-06-27 07:31:36 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.180.134, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:31:42 INFO root.inject_channels:476 - MDTProducer ip: 198.18.93.197
2025-06-27 07:31:42 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.93.197, network=metanet, testnet=sqa2, role=producer
2025-06-27 07:31:47 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtReceiver -  ['dynamic_channel_config_data', 'group_channel_config_data']
2025-06-27 07:31:47 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:31:52 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtreceiver.mdt-test3' MDT system.
2025-06-27 07:31:53 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtreceiver.mdt-test3'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:31:53 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.groups.mdt-test3' MDT system.
2025-06-27 07:31:55 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.groups.mdt-test3'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow'
2025-06-27 07:31:55 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'dynamic_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d959e3190>, 'group_channel_config_data': <python_mdt.channel.mdtchannel.MdtChannel object at 0x7f8d959e34f0>}
2025-06-27 07:31:55 INFO root.inject_channels:484 - MDTServer ip: 198.18.93.176
2025-06-27 07:31:55 INFO root.inject_channels:484 - MDTServer ip: 198.18.101.99
2025-06-27 07:31:55 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.101.99, network=metanet, testnet=sqa2, role=server
2025-06-27 07:32:00 INFO root.inject_channels:484 - MDTServer ip: 198.18.88.154
2025-06-27 07:32:00 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.88.154, network=metanet, testnet=sqa2, role=server
2025-06-27 07:32:05 DEBUG root.mdtserver_machines_fixture:221 - Initializing mdtserver_machines fixture
2025-06-27 07:32:05 DEBUG root.mdtproducer_machines_fixture:239 - Initializing mdtproducer_machines fixture
2025-06-27 07:32:05 INFO root.verify_testnet_fixture:323 - Skipping testnet validation
2025-06-27 07:32:05 INFO conftest.fixture_mdtserver_machine:32 - Selected 198.18.93.176 as the mdtreceiver under test.
2025-06-27 07:32:12 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'ls /root/backup/usr/local/akamai/mdtreceiver/etc/mdt.config.groups*.bak' as root
2025-06-27 07:32:13 WARNING python_alsi.machine.backup_file:509 - WARNING - previously backed up version of /usr/local/akamai/mdtreceiver/etc/mdt.config.groups found in /root/backup/usr/local/akamai/mdtreceiver/etc
2025-06-27 07:32:18 DEBUG python_alsi.machine.backup_file:521 - Backing up file /usr/local/akamai/mdtreceiver/etc/mdt.config.groups to /root/backup/usr/local/akamai/mdtreceiver/etc/mdt.config.groups.2xyC.bak on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:32:29 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'ls /root/backup/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver*.bak' as root
2025-06-27 07:32:31 WARNING python_alsi.machine.backup_file:509 - WARNING - previously backed up version of /usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver found in /root/backup/usr/local/akamai/mdtreceiver/etc
2025-06-27 07:32:36 DEBUG python_alsi.machine.backup_file:521 - Backing up file /usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver to /root/backup/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver.1POS.bak on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
-------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------
2025-06-27 07:32:39 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_default_limits_are_set_in_mdtreceiver_config' ====================
2025-06-27 07:32:39 INFO test_channel_limits_new.test_default_limits_are_set_in_mdtreceiver_config:178 - Checking the default channel config settings on MdtReceiver 198.18.93.176.
2025-06-27 07:32:39 INFO root.pytest_runtest_makereport:52 - Test 'test_default_limits_are_set_in_mdtreceiver_config' passed
PASSED                                                                                                                                                                                                      [ 16%]
------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------
2025-06-27 07:32:39 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_default_limits_are_set_in_mdtreceiver_config -> METANET-2537
tests/mdtreceiver/test_channel_limits_new.py::test_limits_from_granular_section_are_enforced_for_file_channel
------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------
2025-06-27 07:32:39 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
-------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------
2025-06-27 07:32:43 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_from_granular_section_are_enforced_for_file_channel' ====================
2025-06-27 07:32:43 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_1.*'
2025-06-27 07:32:43 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_1.*' in 'mdt-test3' MDT system.
2025-06-27 07:32:45 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_1.*
2025-06-27 07:32:49 DEBUG python_alsi.service.get_local_tableprov_data:414 - Found matching row for machine 198.18.93.176 in table mdtreceiver_status: {'machineip': '198.18.93.176', 'version': '2.22', 'reporting_time': '1751009545', 'client_connection_limit': '110', 'client_connections': '31', 'max_client_connections_1h': '31', 'heap_used': '74435864', 'heap_committed': '150994944', 'heap_max': '12884901888', 'nonheap_used': '97399752', 'nonheap_committed': '110886912', 'bad_zk_state_1h': '0', 'mdtzookeeper_quorum': 'mdtzookeeper_test3.sqa2.qa.akamai.com', 'zk_op_successes_1m': '0', 'zk_op_failures_1m': '0', 'zk_op_errors_1m': '0', 'zk_op_successes_5m': '0', 'zk_op_failures_5m': '0', 'zk_op_errors_5m': '0', 'zk_op_successes_15m': '0', 'zk_op_failures_15m': '0', 'zk_op_errors_15m': '0'}
2025-06-27 07:32:49 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'source /usr/local/akamai/etc/akamai.conf; $MDT_JAVA_HOME/bin/keytool -importkeystore -srckeystore /usr/local/akamai/mdtreceiver/etc/keys/api.jks -srcstorepass password -destkeystore /tmp/python-mdt-secrets/api.p12.tmp -deststorepass password -deststoretype pkcs12 -noprompt -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /a/mdtreceiver/lib/bc-fips.jar -srcstoretype BCFKS' as root
2025-06-27 07:32:53 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'openssl pkcs12 -in /tmp/python-mdt-secrets/api.p12 -nodes -nocerts -out /tmp/python-mdt-secrets/api.private_key.tmp -passin pass:password' as root
2025-06-27 07:32:55 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'openssl pkcs12 -in /tmp/python-mdt-secrets/api.p12 -nodes -nokeys -out /tmp/python-mdt-secrets/api.certificate.tmp -passin pass:password' as root
2025-06-27 07:32:57 INFO test_channel_limits_new.test_limits_from_granular_section_are_enforced_for_file_channel:204 - Setting 'max_channel_mb' to 2 MB for channel test_mdt-test3.multi_1.W4Ll.
2025-06-27 07:32:57 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:33:02 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:33:05 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:33:10 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:33:12 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009588.mdti2q-g
2025-06-27 07:33:17 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:33:19 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009588.mdti2q-g
2025-06-27 07:33:25 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:33:26 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009589 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:33:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009589
2025-06-27 07:33:41 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1750932290, "numBadAcks": 0, "mdtTimestamp": 1750932222, "mdtTimestampMillisPart": 0}
2025-06-27 07:33:41 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009589 or higher (current=1750932222); sleeping 5s (600s remaining)
2025-06-27 07:33:57 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0}
2025-06-27 07:33:57 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009589
2025-06-27 07:33:57 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009589 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:33:57 INFO test_channel_limits_new._prepare_content:45 - Preparing channel test_mdt-test3.multi_1.W4Ll with file of size 3145728 bytes.
2025-06-27 07:33:57 INFO python_mdt.channel.content.generate_random_file_content:163 - Generating a 3145728 byte random text file
2025-06-27 07:33:57 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_1.W4Ll with 1 files of size 3145728 bytes.
2025-06-27 07:34:00 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_1.W4Ll?zone=1&submitType=full
2025-06-27 07:34:07 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_1.W4Ll
2025-06-27 07:34:08 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_1.W4Ll.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded channel size limit","status":413,"detail":"Submission size exceeds the allowed limit of 2097152 bytes","requestID":"REQID-a9907b0e-3f0e-4d55-a8f4-17c7631e3c89-mdt-test-8"}')
2025-06-27 07:34:08 INFO test_channel_limits_new._prepare_content:45 - Preparing channel test_mdt-test3.multi_1.W4Ll with file of size 1048576 bytes.
2025-06-27 07:34:08 INFO python_mdt.channel.content.generate_random_file_content:163 - Generating a 1048576 byte random text file
2025-06-27 07:34:08 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_1.W4Ll with 1 files of size 1048576 bytes.
2025-06-27 07:34:09 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_1.W4Ll?zone=1&submitType=full
2025-06-27 07:34:14 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_1.W4Ll
2025-06-27 07:34:15 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_1.W4Ll with in channel limits is succeeded.
2025-06-27 07:34:15 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_from_granular_section_are_enforced_for_file_channel' passed
PASSED                                                                                                                                                                                                      [ 33%]
------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------
2025-06-27 07:34:15 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_from_granular_section_are_enforced_for_file_channel -> METANET-2543
2025-06-27 07:34:15 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation
2025-06-27 07:34:15 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:34:17 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:34:22 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:34:24 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009660.mdti2q-g
2025-06-27 07:34:29 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:34:31 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009660.mdti2q-g
2025-06-27 07:34:37 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:34:38 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009660 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:34:38 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009660
2025-06-27 07:34:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0}
2025-06-27 07:34:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (600s remaining)
2025-06-27 07:35:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0}
2025-06-27 07:35:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (595s remaining)
2025-06-27 07:35:21 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0}
2025-06-27 07:35:21 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (590s remaining)
2025-06-27 07:35:37 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0}
2025-06-27 07:35:37 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (585s remaining)
2025-06-27 07:35:53 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0}
2025-06-27 07:35:53 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (580s remaining)
2025-06-27 07:36:09 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009753, "numBadAcks": 0, "mdtTimestamp": 1751009660, "mdtTimestampMillisPart": 0}
2025-06-27 07:36:09 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009660
2025-06-27 07:36:09 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009660 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
tests/mdtreceiver/test_channel_limits_new.py::test_limits_for_ns_are_enforced_for_dir_channel
------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------
2025-06-27 07:36:09 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
-------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------
2025-06-27 07:36:13 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_for_ns_are_enforced_for_dir_channel' ====================
2025-06-27 07:36:13 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*'
2025-06-27 07:36:13 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system.
2025-06-27 07:36:15 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.*
2025-06-27 07:36:15 INFO test_channel_limits_new.test_limits_for_ns_are_enforced_for_dir_channel:226 - Setting limits {'max_channel_file_count': 10, 'max_channel_mb': 20} for namespace test_mdt-test3.multi_2.dir.*.
2025-06-27 07:36:15 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:36:19 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:36:21 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:36:26 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:36:28 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009784.mdti2q-g
2025-06-27 07:36:33 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:36:35 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009784.mdti2q-g
2025-06-27 07:36:40 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:36:40 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009784 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:36:40 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009784
2025-06-27 07:36:51 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009753, "numBadAcks": 0, "mdtTimestamp": 1751009660, "mdtTimestampMillisPart": 0}
2025-06-27 07:36:51 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009784 or higher (current=1751009660); sleeping 5s (600s remaining)
2025-06-27 07:37:07 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009753, "numBadAcks": 0, "mdtTimestamp": 1751009660, "mdtTimestampMillisPart": 0}
2025-06-27 07:37:07 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009784 or higher (current=1751009660); sleeping 5s (595s remaining)
2025-06-27 07:37:23 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009753, "numBadAcks": 0, "mdtTimestamp": 1751009660, "mdtTimestampMillisPart": 0}
2025-06-27 07:37:23 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009784 or higher (current=1751009660); sleeping 5s (590s remaining)
2025-06-27 07:37:39 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3c20f2dd1fcf746e1c690b60a183d2aa4ef4d12907dbe2025dea59617a5a7365", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009856, "numBadAcks": 0, "mdtTimestamp": 1751009784, "mdtTimestampMillisPart": 0}
2025-06-27 07:37:39 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009784
2025-06-27 07:37:39 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009784 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:37:39 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.0vWN with 15 files of size 1048576 bytes.
2025-06-27 07:37:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:37:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file
2025-06-27 07:37:40 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.0vWN with 15 files of size 1048576 bytes.
2025-06-27 07:37:42 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN?zone=1&submitType=full
2025-06-27 07:37:51 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN
2025-06-27 07:37:52 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.0vWN.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 15, exceeding the configured limit of 10","requestID":"REQID-116d6386-e530-4651-b97f-84fab09ffbb4-mdt-test-8"}')
2025-06-27 07:37:52 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.0vWN with 5 files of size 5242880 bytes.
2025-06-27 07:37:52 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 5242880 byte random text file
2025-06-27 07:37:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 5242880 byte random text file
2025-06-27 07:37:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 5242880 byte random text file
2025-06-27 07:37:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 5242880 byte random text file
2025-06-27 07:37:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 5242880 byte random text file
2025-06-27 07:37:54 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.0vWN with 5 files of size 5242880 bytes.
2025-06-27 07:37:55 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN?zone=1&submitType=full
2025-06-27 07:38:06 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN
2025-06-27 07:38:06 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.0vWN.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded channel size limit","status":413,"detail":"Submission size exceeds the allowed limit of 20971520 bytes","requestID":"REQID-bc23c06e-968e-4355-8ebe-238d19bee25f-mdt-test-8"}')
2025-06-27 07:38:06 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.0vWN with 10 files of size 1048576 bytes.
2025-06-27 07:38:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:38:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file
2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file
2025-06-27 07:38:07 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.0vWN with 10 files of size 1048576 bytes.
2025-06-27 07:38:09 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN?zone=1&submitType=full
2025-06-27 07:38:17 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN
2025-06-27 07:38:17 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.0vWN with in channel limits is succeeded.
2025-06-27 07:38:17 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_for_ns_are_enforced_for_dir_channel' passed
PASSED                                                                                                                                                                                                      [ 50%]
------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------
2025-06-27 07:38:17 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_for_ns_are_enforced_for_dir_channel -> METANET-2538
2025-06-27 07:38:17 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation
2025-06-27 07:38:17 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:38:19 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:38:24 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:38:26 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009902.mdti2q-g
2025-06-27 07:38:31 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:38:33 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009902.mdti2q-g
2025-06-27 07:38:37 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:38:38 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009902 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:38:38 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009902
2025-06-27 07:38:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3c20f2dd1fcf746e1c690b60a183d2aa4ef4d12907dbe2025dea59617a5a7365", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009856, "numBadAcks": 0, "mdtTimestamp": 1751009784, "mdtTimestampMillisPart": 0}
2025-06-27 07:38:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009902 or higher (current=1751009784); sleeping 5s (600s remaining)
2025-06-27 07:39:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3c20f2dd1fcf746e1c690b60a183d2aa4ef4d12907dbe2025dea59617a5a7365", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009856, "numBadAcks": 0, "mdtTimestamp": 1751009784, "mdtTimestampMillisPart": 0}
2025-06-27 07:39:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009902 or higher (current=1751009784); sleeping 5s (595s remaining)
2025-06-27 07:39:21 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3c20f2dd1fcf746e1c690b60a183d2aa4ef4d12907dbe2025dea59617a5a7365", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009856, "numBadAcks": 0, "mdtTimestamp": 1751009784, "mdtTimestampMillisPart": 0}
2025-06-27 07:39:21 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009902 or higher (current=1751009784); sleeping 5s (590s remaining)
2025-06-27 07:39:37 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009969, "numBadAcks": 0, "mdtTimestamp": 1751009902, "mdtTimestampMillisPart": 0}
2025-06-27 07:39:37 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009902
2025-06-27 07:39:37 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009902 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
tests/mdtreceiver/test_channel_limits_new.py::test_limits_are_enforced_on_resulting_channel_state
------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------
2025-06-27 07:39:37 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
-------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------
2025-06-27 07:39:41 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_are_enforced_on_resulting_channel_state' ====================
2025-06-27 07:39:41 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*'
2025-06-27 07:39:41 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system.
2025-06-27 07:39:43 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.*
2025-06-27 07:39:43 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:248 - Setting limits {'max_channel_file_count': 10, 'max_channel_mb': 20} for namespace test_mdt-test3.multi_2.dir.gZGP.
2025-06-27 07:39:43 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:39:47 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:39:49 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:39:54 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:39:56 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009992.mdti2q-g
2025-06-27 07:40:00 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:40:02 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009992.mdti2q-g
2025-06-27 07:40:07 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:40:07 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009992 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:40:07 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009992
2025-06-27 07:40:18 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "22baebe4e376045da6704aa211abdaf9555620a20f65632e0b001408176d9db4", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010012, "numBadAcks": 0, "mdtTimestamp": 1751009992, "mdtTimestampMillisPart": 0}
2025-06-27 07:40:18 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009992
2025-06-27 07:40:18 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009992 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:40:18 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.gZGP with 5 files of size 1048576 bytes.
2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:40:18 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.gZGP with 5 files of size 1048576 bytes.
2025-06-27 07:40:20 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP?zone=1&submitType=full
2025-06-27 07:40:27 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP
2025-06-27 07:40:27 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.gZGP with in channel limits is succeeded.
2025-06-27 07:40:27 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:257 - Submitting content such that the resulting channel state exceeds the channel size limit.
2025-06-27 07:40:27 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.gZGP with 4 files of size 4194304 bytes.
2025-06-27 07:40:28 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 4194304 byte random text file
2025-06-27 07:40:28 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 4194304 byte random text file
2025-06-27 07:40:28 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 4194304 byte random text file
2025-06-27 07:40:28 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 4194304 byte random text file
2025-06-27 07:40:30 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP?zone=1&submitType=incr_lenient
2025-06-27 07:40:39 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP
2025-06-27 07:40:39 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:266 - Incremental lenient submission for channel test_mdt-test3.multi_2.dir.gZGP.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded channel size limit","status":413,"detail":"Operation will result in a channel size of 22020096 bytes, exceeding the configured limit of 20971520 bytes","requestID":"REQID-e290595d-35a1-4576-9aee-01371271375d-mdt-test-8"}'
2025-06-27 07:40:39 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:257 - Submitting content such that the resulting channel state exceeds the file count limit.
2025-06-27 07:40:39 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.gZGP with 6 files of size 1048576 bytes.
2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file
2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file
2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file
2025-06-27 07:40:41 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP?zone=1&submitType=incr_lenient
2025-06-27 07:40:47 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP
2025-06-27 07:40:47 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:266 - Incremental lenient submission for channel test_mdt-test3.multi_2.dir.gZGP.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 11, exceeding the configured limit of 10","requestID":"REQID-4477923d-3098-4062-84aa-20a4ff922e7b-mdt-test-8"}'
2025-06-27 07:40:47 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_are_enforced_on_resulting_channel_state' passed
PASSED                                                                                                                                                                                                      [ 66%]
------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------
2025-06-27 07:40:47 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_are_enforced_on_resulting_channel_state -> METANET-2540
2025-06-27 07:40:47 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation
2025-06-27 07:40:47 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:40:49 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:40:53 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:40:55 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010052.mdti2q-g
2025-06-27 07:41:00 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:41:02 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010052.mdti2q-g
2025-06-27 07:41:06 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:41:07 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010052 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:41:07 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010052
2025-06-27 07:41:17 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "22baebe4e376045da6704aa211abdaf9555620a20f65632e0b001408176d9db4", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010012, "numBadAcks": 0, "mdtTimestamp": 1751009992, "mdtTimestampMillisPart": 0}
2025-06-27 07:41:17 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010052 or higher (current=1751009992); sleeping 5s (600s remaining)
2025-06-27 07:41:33 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "22baebe4e376045da6704aa211abdaf9555620a20f65632e0b001408176d9db4", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010012, "numBadAcks": 0, "mdtTimestamp": 1751009992, "mdtTimestampMillisPart": 0}
2025-06-27 07:41:33 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010052 or higher (current=1751009992); sleeping 5s (595s remaining)
2025-06-27 07:41:48 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010093, "numBadAcks": 0, "mdtTimestamp": 1751010052, "mdtTimestampMillisPart": 0}
2025-06-27 07:41:48 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010052
2025-06-27 07:41:48 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010052 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
tests/mdtreceiver/test_channel_limits_new.py::test_limits_can_be_set_at_subnamespace
------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------
2025-06-27 07:41:48 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
-------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------
2025-06-27 07:41:53 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_can_be_set_at_subnamespace' ====================
2025-06-27 07:41:53 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:41:57 INFO test_channel_limits_new._test_runner:131 - Setting channel limits for multiple channels/namespaces.
2025-06-27 07:41:57 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*'
2025-06-27 07:41:57 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system.
2025-06-27 07:41:58 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.*
2025-06-27 07:41:58 INFO test_channel_limits_new._test_runner:135 - Test channel is test_mdt-test3.multi_2.dir.postfix1.channel1.
2025-06-27 07:41:58 INFO test_channel_limits_new._helper_to_update_granular_config:113 - Setting limits {'max_channel_file_count': 10, 'max_channel_mb': 20} for test_mdt-test3.multi_2.dir.postfix1.*.
2025-06-27 07:41:58 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*'
2025-06-27 07:41:58 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system.
2025-06-27 07:42:00 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.*
2025-06-27 07:42:00 INFO test_channel_limits_new._test_runner:135 - Test channel is test_mdt-test3.multi_2.dir.postfix2.channel2.
2025-06-27 07:42:00 INFO test_channel_limits_new._helper_to_update_granular_config:113 - Setting limits {'max_channel_file_count': 20, 'max_channel_mb': 30} for test_mdt-test3.multi_2.dir.postfix2.*.
2025-06-27 07:42:00 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:42:01 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:42:06 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:42:08 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010124.mdti2q-g
2025-06-27 07:42:12 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:42:14 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010124.mdti2q-g
2025-06-27 07:42:19 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:42:19 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010125 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:42:19 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010125
2025-06-27 07:42:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010093, "numBadAcks": 0, "mdtTimestamp": 1751010052, "mdtTimestampMillisPart": 0}
2025-06-27 07:42:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010125 or higher (current=1751010052); sleeping 5s (600s remaining)
2025-06-27 07:42:45 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010093, "numBadAcks": 0, "mdtTimestamp": 1751010052, "mdtTimestampMillisPart": 0}
2025-06-27 07:42:45 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010125 or higher (current=1751010052); sleeping 5s (595s remaining)
2025-06-27 07:43:01 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010093, "numBadAcks": 0, "mdtTimestamp": 1751010052, "mdtTimestampMillisPart": 0}
2025-06-27 07:43:01 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010125 or higher (current=1751010052); sleeping 5s (590s remaining)
2025-06-27 07:43:18 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "818ae1449aded5cbf112ec9aa03f4a109140fb325be523e0252a9668bf59d451", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010186, "numBadAcks": 0, "mdtTimestamp": 1751010125, "mdtTimestampMillisPart": 0}
2025-06-27 07:43:18 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010125
2025-06-27 07:43:18 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010125 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:43:18 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.postfix1.channel1 with 15 files of size 1048576 bytes.
2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file
2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file
2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file
2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file
2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file
2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file
2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file
2025-06-27 07:43:19 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.postfix1.channel1 with 15 files of size 1048576 bytes.
2025-06-27 07:43:21 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix1.channel1?zone=1&submitType=full
2025-06-27 07:43:29 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix1.channel1
2025-06-27 07:43:30 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.postfix1.channel1.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 15, exceeding the configured limit of 10","requestID":"REQID-f9a2b7c0-001a-4825-9880-c36a85eea96b-mdt-test-8"}')
2025-06-27 07:43:30 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.postfix2.channel2 with 25 files of size 1048576 bytes.
2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_15.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_16.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_17.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_18.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_19.txt as a 1048576 byte random text file
2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_20.txt as a 1048576 byte random text file
2025-06-27 07:43:32 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_21.txt as a 1048576 byte random text file
2025-06-27 07:43:32 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_22.txt as a 1048576 byte random text file
2025-06-27 07:43:32 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_23.txt as a 1048576 byte random text file
2025-06-27 07:43:32 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_24.txt as a 1048576 byte random text file
2025-06-27 07:43:32 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.postfix2.channel2 with 25 files of size 1048576 bytes.
2025-06-27 07:43:33 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix2.channel2?zone=1&submitType=full
2025-06-27 07:43:43 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix2.channel2
2025-06-27 07:43:44 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.postfix2.channel2.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 25, exceeding the configured limit of 20","requestID":"REQID-c6355a08-f538-4eba-be50-c68a1842f471-mdt-test-8"}')
2025-06-27 07:43:44 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.postfix1.channel1 with 8 files of size 1048576 bytes.
2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:43:44 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.postfix1.channel1 with 8 files of size 1048576 bytes.
2025-06-27 07:43:46 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix1.channel1?zone=1&submitType=full
2025-06-27 07:43:53 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix1.channel1
2025-06-27 07:43:53 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.postfix1.channel1 with in channel limits is succeeded.
2025-06-27 07:43:53 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.postfix2.channel2 with 15 files of size 1048576 bytes.
2025-06-27 07:43:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:43:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:43:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:43:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file
2025-06-27 07:43:54 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.postfix2.channel2 with 15 files of size 1048576 bytes.
2025-06-27 07:43:56 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix2.channel2?zone=1&submitType=full
2025-06-27 07:44:04 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix2.channel2
2025-06-27 07:44:04 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.postfix2.channel2 with in channel limits is succeeded.
2025-06-27 07:44:04 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_can_be_set_at_subnamespace' passed
PASSED                                                                                                                                                                                                      [ 83%]
------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------
2025-06-27 07:44:04 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_can_be_set_at_subnamespace -> METANET-2542
2025-06-27 07:44:04 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation
2025-06-27 07:44:04 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:44:06 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:44:12 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:44:14 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010249.mdti2q-g
2025-06-27 07:44:18 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:44:20 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010249.mdti2q-g
2025-06-27 07:44:25 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:44:25 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010249 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:44:25 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010249
2025-06-27 07:44:36 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "818ae1449aded5cbf112ec9aa03f4a109140fb325be523e0252a9668bf59d451", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010186, "numBadAcks": 0, "mdtTimestamp": 1751010125, "mdtTimestampMillisPart": 0}
2025-06-27 07:44:36 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010249 or higher (current=1751010125); sleeping 5s (600s remaining)
2025-06-27 07:44:51 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010281, "numBadAcks": 0, "mdtTimestamp": 1751010249, "mdtTimestampMillisPart": 0}
2025-06-27 07:44:51 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010249
2025-06-27 07:44:51 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010249 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
tests/mdtreceiver/test_channel_limits_new.py::test_limits_can_be_set_per_channel
------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------
2025-06-27 07:44:51 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
-------------------------------------------------------------------------------------------------- live log call --------------------------------------------------------------------------------------------------
2025-06-27 07:44:55 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_can_be_set_per_channel' ====================
2025-06-27 07:44:55 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:45:00 INFO test_channel_limits_new._test_runner:131 - Setting channel limits for multiple channels/namespaces.
2025-06-27 07:45:00 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*'
2025-06-27 07:45:00 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system.
2025-06-27 07:45:01 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.*
2025-06-27 07:45:01 INFO test_channel_limits_new._test_runner:135 - Test channel is test_mdt-test3.multi_2.dir.NbM8.
2025-06-27 07:45:01 INFO test_channel_limits_new._helper_to_update_granular_config:113 - Setting limits {'max_channel_file_count': 10, 'max_channel_mb': 20} for test_mdt-test3.multi_2.dir.NbM8.
2025-06-27 07:45:01 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*'
2025-06-27 07:45:01 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system.
2025-06-27 07:45:02 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.*
2025-06-27 07:45:02 INFO test_channel_limits_new._test_runner:135 - Test channel is test_mdt-test3.multi_2.dir.1RXG.
2025-06-27 07:45:02 INFO test_channel_limits_new._helper_to_update_granular_config:113 - Setting limits {'max_channel_file_count': 20, 'max_channel_mb': 30} for test_mdt-test3.multi_2.dir.1RXG.
2025-06-27 07:45:02 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:45:04 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:45:09 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:45:11 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010307.mdti2q-g
2025-06-27 07:45:16 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:45:18 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010307.mdti2q-g
2025-06-27 07:45:23 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:45:23 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010307 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:45:23 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010307
2025-06-27 07:45:34 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010281, "numBadAcks": 0, "mdtTimestamp": 1751010249, "mdtTimestampMillisPart": 0}
2025-06-27 07:45:34 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010307 or higher (current=1751010249); sleeping 5s (600s remaining)
2025-06-27 07:45:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010281, "numBadAcks": 0, "mdtTimestamp": 1751010249, "mdtTimestampMillisPart": 0}
2025-06-27 07:45:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010307 or higher (current=1751010249); sleeping 5s (595s remaining)
2025-06-27 07:46:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "5b0c2c8b547864d6532256036a549f3989f7be8fbcda026ba1d159b53d92d8cf", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010363, "numBadAcks": 0, "mdtTimestamp": 1751010307, "mdtTimestampMillisPart": 0}
2025-06-27 07:46:05 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010307
2025-06-27 07:46:05 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010307 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:46:05 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.NbM8 with 15 files of size 1048576 bytes.
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file
2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file
2025-06-27 07:46:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file
2025-06-27 07:46:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file
2025-06-27 07:46:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file
2025-06-27 07:46:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file
2025-06-27 07:46:06 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.NbM8 with 15 files of size 1048576 bytes.
2025-06-27 07:46:07 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.NbM8?zone=1&submitType=full
2025-06-27 07:46:16 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.NbM8
2025-06-27 07:46:17 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.NbM8.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 15, exceeding the configured limit of 10","requestID":"REQID-c7c937f4-22b4-4bdd-bfb0-4bbe1d037be5-mdt-test-8"}')
2025-06-27 07:46:17 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.1RXG with 25 files of size 1048576 bytes.
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file
2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_15.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_16.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_17.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_18.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_19.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_20.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_21.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_22.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_23.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_24.txt as a 1048576 byte random text file
2025-06-27 07:46:18 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.1RXG with 25 files of size 1048576 bytes.
2025-06-27 07:46:20 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.1RXG?zone=1&submitType=full
2025-06-27 07:46:30 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.1RXG
2025-06-27 07:46:31 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.1RXG.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 25, exceeding the configured limit of 20","requestID":"REQID-a0ae7dcf-c891-4827-a41e-b0a4e8e6643e-mdt-test-8"}')
2025-06-27 07:46:31 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.NbM8 with 8 files of size 1048576 bytes.
2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:46:31 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.NbM8 with 8 files of size 1048576 bytes.
2025-06-27 07:46:33 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.NbM8?zone=1&submitType=full
2025-06-27 07:46:40 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.NbM8
2025-06-27 07:46:40 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.NbM8 with in channel limits is succeeded.
2025-06-27 07:46:40 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.1RXG with 15 files of size 1048576 bytes.
2025-06-27 07:46:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file
2025-06-27 07:46:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file
2025-06-27 07:46:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file
2025-06-27 07:46:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file
2025-06-27 07:46:41 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.1RXG with 15 files of size 1048576 bytes.
2025-06-27 07:46:43 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.1RXG?zone=1&submitType=full
2025-06-27 07:46:52 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.1RXG
2025-06-27 07:46:53 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.1RXG with in channel limits is succeeded.
2025-06-27 07:46:53 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_can_be_set_per_channel' passed
PASSED                                                                                                                                                                                                      [100%]
------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------
2025-06-27 07:46:53 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_can_be_set_per_channel -> METANET-2539
2025-06-27 07:46:53 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation
2025-06-27 07:46:53 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:46:54 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:46:59 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:47:01 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010418.mdti2q-g
2025-06-27 07:47:06 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:47:08 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010418.mdti2q-g
2025-06-27 07:47:13 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:47:14 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010418 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:47:14 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010418
2025-06-27 07:47:24 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "5b0c2c8b547864d6532256036a549f3989f7be8fbcda026ba1d159b53d92d8cf", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010363, "numBadAcks": 0, "mdtTimestamp": 1751010307, "mdtTimestampMillisPart": 0}
2025-06-27 07:47:24 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010418 or higher (current=1751010307); sleeping 5s (600s remaining)
2025-06-27 07:47:40 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0}
2025-06-27 07:47:40 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010418
2025-06-27 07:47:40 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010418 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:47:45 INFO src.mdt.utils.backup_restore_files:139 - Restoring 'mdt.config.groups.mdt-test3'.
{
  "granular_config": {
    "global_config": {
      "submit_to_zk": true
    }
  }
}
2025-06-27 07:47:45 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3'
2025-06-27 07:47:46 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full
2025-06-27 07:47:51 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:47:53 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010469.mdti2q-g
2025-06-27 07:47:57 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:47:59 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010469.mdti2q-g
2025-06-27 07:48:04 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3
2025-06-27 07:48:04 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010469 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:48:04 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010469
2025-06-27 07:48:15 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0}
2025-06-27 07:48:15 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010469 or higher (current=1751010418); sleeping 5s (600s remaining)
2025-06-27 07:48:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0}
2025-06-27 07:48:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010469 or higher (current=1751010418); sleeping 5s (595s remaining)
2025-06-27 07:48:46 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0}
2025-06-27 07:48:46 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010469 or higher (current=1751010418); sleeping 5s (590s remaining)
2025-06-27 07:49:01 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0}
2025-06-27 07:49:01 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010469 or higher (current=1751010418); sleeping 5s (585s remaining)
2025-06-27 07:49:18 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010550, "numBadAcks": 0, "mdtTimestamp": 1751010469, "mdtTimestampMillisPart": 0}
2025-06-27 07:49:18 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010469
2025-06-27 07:49:18 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010469 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:49:23 INFO src.mdt.utils.backup_restore_files:139 - Restoring 'mdt.config.mdtreceiver.mdt-test3'.
{
  "enabled_jetstream_systems": [
    "freeflow"
  ],
  "locking_installed_everywhere": false,
  "write_locking_mode": "ZK"
}
2025-06-27 07:49:23 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.mdtreceiver.mdt-test3'
2025-06-27 07:49:25 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3?zone=1&submitType=full
2025-06-27 07:49:29 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3
2025-06-27 07:49:31 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3?zone=2&uuid=mdt.config.mdtreceiver.mdt-test3.1.0000001751010568.mdti2q-g
2025-06-27 07:49:36 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3
2025-06-27 07:49:38 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3?zone=3&uuid=mdt.config.mdtreceiver.mdt-test3.1.0000001751010568.mdti2q-g
2025-06-27 07:49:42 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3
2025-06-27 07:49:43 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.mdtreceiver.mdt-test3 has been updated to timestamp 1751010568 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:49:43 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver' for channel 'mdt.config.mdtreceiver.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010568
2025-06-27 07:49:54 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.mdtreceiver.mdt-test3' is: {"sha256": "3689d7b7904acde8f3f9778f2ef2040d8966f8508910c42e2ce8b5a983b6cb78", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver", "contentVersion": "1.3", "channel": "mdt.config.mdtreceiver.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1750932299, "numBadAcks": 0, "mdtTimestamp": 1750932276, "mdtTimestampMillisPart": 0}
2025-06-27 07:49:54 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.mdtreceiver.mdt-test3 has not yet been updated to the requested timestamp 1751010568 or higher (current=1750932276); sleeping 5s (600s remaining)
2025-06-27 07:50:09 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.mdtreceiver.mdt-test3' is: {"sha256": "3689d7b7904acde8f3f9778f2ef2040d8966f8508910c42e2ce8b5a983b6cb78", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver", "contentVersion": "1.3", "channel": "mdt.config.mdtreceiver.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1750932299, "numBadAcks": 0, "mdtTimestamp": 1750932276, "mdtTimestampMillisPart": 0}
2025-06-27 07:50:09 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.mdtreceiver.mdt-test3 has not yet been updated to the requested timestamp 1751010568 or higher (current=1750932276); sleeping 5s (595s remaining)
2025-06-27 07:50:25 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.mdtreceiver.mdt-test3' is: {"sha256": "3689d7b7904acde8f3f9778f2ef2040d8966f8508910c42e2ce8b5a983b6cb78", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver", "contentVersion": "1.3", "channel": "mdt.config.mdtreceiver.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010612, "numBadAcks": 0, "mdtTimestamp": 1751010568, "mdtTimestampMillisPart": 0}
2025-06-27 07:50:25 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.mdtreceiver.mdt-test3 is up to date with requested timestamp 1751010568
2025-06-27 07:50:25 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.mdtreceiver.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010568 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server
2025-06-27 07:50:25 DEBUG root.delete_paths:344 - Deleting temporary paths.
========================================================================================= 6 passed in 1721.82s (0:28:41) ==========================================================================================
Filename: None. Size: 242kb. View raw, , hex, or download this file.

This paste expires on 2025-07-04 07:50:27.493919. Pasted through deprecated-web.