=============================================================================================== test session starts =============================================================================================== platform linux -- Python 3.10.14, pytest-8.3.5, pluggy-1.5.0 -- /home/rjagarla/work/pytest-automation/new_runner/bin/python rootdir: /home/rjagarla/work/pytest-automation/mdt-automation configfile: pytest.ini plugins: jira-0.3.21, mdt-2.1.5.dev21+gf5e1951, alsi-3.1.4 collecting ... collected 6 items tests/mdtreceiver/test_channel_limits_new.py::test_default_limits_are_set_in_mdtreceiver_config ------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------- 2025-06-27 07:21:43 DEBUG root.mdt_network_inventory_fixture:119 - Adding 'metanet' to inventory. 2025-06-27 07:21:43 INFO python_alsi.testnet.networks:272 - Loading networks from testnet sqa2. 2025-06-27 07:21:43 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=,name=freeflow,fetcher= 2025-06-27 07:21:54 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=,name=infra,fetcher= 2025-06-27 07:21:57 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=,name=ff,fetcher= 2025-06-27 07:22:01 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=,name=ness,fetcher= 2025-06-27 07:22:05 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=,name=essl,fetcher= 2025-06-27 07:22:08 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=,name=volta,fetcher= 2025-06-27 07:22:12 DEBUG python_alsi.util.__call__:474 - Creating a new instance of DirectNetwork for key testnet=,name=metanet,fetcher= 2025-06-27 07:22:15 INFO python_alsi.testnet.networks:277 - Loaded 7 networks from testnet sqa2. 2025-06-27 07:22:15 DEBUG python_mdt.system.mdtsystem._system_info_setup:178 - Setting up MDT system: 'freeflow' in Network: 'metanet' 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=,ip=198.19.8.7 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.8.7,nat_gateway=None,authgate=None 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=,ip=198.19.3.90 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.3.90,nat_gateway=None,authgate=None 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=,ip=198.19.11.120 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.11.120,nat_gateway=None,authgate=None 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=,ip=198.19.16.14 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.16.14,nat_gateway=None,authgate=None 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=,ip=198.18.101.101 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.101.101,nat_gateway=None,authgate=None 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of NetworkMachine for key network=,ip=198.18.101.102 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.101.102,nat_gateway=None,authgate=None 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=,ip=198.18.116.171 2025-06-27 07:22:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.171,nat_gateway=None,authgate=None 2025-06-27 07:22:16 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.171] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:22:16 INFO python_alsi.connection.connection:168 - [198.18.116.171] Connecting as user root 2025-06-27 07:22:16 DEBUG python_alsi.connection.connection:193 - [198.18.116.171] Connecting directly using ssh 2025-06-27 07:22:16 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.171'] 2025-06-27 07:22:19 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.171 2025-06-27 07:22:20 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:22:20 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:22:21 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.171', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:22:24 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:22:26 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.171] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:22:28 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:22:28 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:22:28 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=,ip=198.18.116.174 2025-06-27 07:22:28 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.174,nat_gateway=None,authgate=None 2025-06-27 07:22:28 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.174] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:22:28 INFO python_alsi.connection.connection:168 - [198.18.116.174] Connecting as user root 2025-06-27 07:22:28 DEBUG python_alsi.connection.connection:193 - [198.18.116.174] Connecting directly using ssh 2025-06-27 07:22:28 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.174'] 2025-06-27 07:22:32 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.174 2025-06-27 07:22:32 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:22:32 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:22:33 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.174', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:22:36 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:22:38 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.174] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:22:40 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:22:40 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:22:40 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.116.177 2025-06-27 07:22:40 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.177,nat_gateway=None,authgate=None 2025-06-27 07:22:40 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.177] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:22:40 INFO python_alsi.connection.connection:168 - [198.18.116.177] Connecting as user root 2025-06-27 07:22:40 DEBUG python_alsi.connection.connection:193 - [198.18.116.177] Connecting directly using ssh 2025-06-27 07:22:40 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.177'] 2025-06-27 07:22:44 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.177 2025-06-27 07:22:44 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:22:44 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:22:45 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.177', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:22:48 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:22:50 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.177] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:22:52 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_3_198.18.116.177.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_3_198.18.116.177.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_3_198.18.116.177.json')} 2025-06-27 07:22:52 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_3_198.18.116.177.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_3_198.18.116.177.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_3_198.18.116.177.json')} 2025-06-27 07:22:52 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.228.115 2025-06-27 07:22:52 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.228.115,nat_gateway=None,authgate=None 2025-06-27 07:22:52 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.115] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:22:52 INFO python_alsi.connection.connection:168 - [198.18.228.115] Connecting as user root 2025-06-27 07:22:52 DEBUG python_alsi.connection.connection:193 - [198.18.228.115] Connecting directly using ssh 2025-06-27 07:22:52 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.115'] 2025-06-27 07:22:56 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.228.115 2025-06-27 07:22:56 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:22:56 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:22:58 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.115', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:23:00 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:23:03 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.115] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:23:04 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_4_198.18.228.115.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_4_198.18.228.115.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_4_198.18.228.115.json')} 2025-06-27 07:23:04 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_4_198.18.228.115.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_4_198.18.228.115.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_4_198.18.228.115.json')} 2025-06-27 07:23:04 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.19.11.121 2025-06-27 07:23:04 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.11.121,nat_gateway=None,authgate=None 2025-06-27 07:23:04 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.121] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:23:04 INFO python_alsi.connection.connection:168 - [198.19.11.121] Connecting as user root 2025-06-27 07:23:04 DEBUG python_alsi.connection.connection:193 - [198.19.11.121] Connecting directly using ssh 2025-06-27 07:23:04 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.121'] 2025-06-27 07:23:08 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.11.121 2025-06-27 07:23:09 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:23:09 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:23:10 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.121', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:23:12 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:23:15 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.121] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:23:16 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:23:16 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:23:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=,ip=198.18.116.172 2025-06-27 07:23:16 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.172,nat_gateway=None,authgate=None 2025-06-27 07:23:16 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.172] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:23:16 INFO python_alsi.connection.connection:168 - [198.18.116.172] Connecting as user root 2025-06-27 07:23:16 DEBUG python_alsi.connection.connection:193 - [198.18.116.172] Connecting directly using ssh 2025-06-27 07:23:16 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.172'] 2025-06-27 07:23:20 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.172 2025-06-27 07:23:21 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:23:21 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:23:22 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.172', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:23:25 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:23:27 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.172] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:23:29 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:23:29 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:23:29 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=,ip=198.18.116.173 2025-06-27 07:23:29 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.173,nat_gateway=None,authgate=None 2025-06-27 07:23:29 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.173] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:23:29 INFO python_alsi.connection.connection:168 - [198.18.116.173] Connecting as user root 2025-06-27 07:23:29 DEBUG python_alsi.connection.connection:193 - [198.18.116.173] Connecting directly using ssh 2025-06-27 07:23:29 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.173'] 2025-06-27 07:23:32 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.173 2025-06-27 07:23:33 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:23:33 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:23:34 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.173', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:23:37 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:23:39 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.173] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:23:41 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:23:41 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:23:41 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtOriginMachine for key network=,ip=198.18.116.178 2025-06-27 07:23:41 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.178,nat_gateway=None,authgate=None 2025-06-27 07:23:41 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.178] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:23:41 INFO python_alsi.connection.connection:168 - [198.18.116.178] Connecting as user root 2025-06-27 07:23:41 DEBUG python_alsi.connection.connection:193 - [198.18.116.178] Connecting directly using ssh 2025-06-27 07:23:41 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.178'] 2025-06-27 07:23:44 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.178 2025-06-27 07:23:45 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:23:45 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:23:46 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.178', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:23:49 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:23:51 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.178] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:23:53 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files {'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None} 2025-06-27 07:23:53 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files {'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None} 2025-06-27 07:23:53 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.116.175 2025-06-27 07:23:53 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.175,nat_gateway=None,authgate=None 2025-06-27 07:23:53 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.175] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:23:53 INFO python_alsi.connection.connection:168 - [198.18.116.175] Connecting as user root 2025-06-27 07:23:53 DEBUG python_alsi.connection.connection:193 - [198.18.116.175] Connecting directly using ssh 2025-06-27 07:23:53 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.175'] 2025-06-27 07:23:57 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.175 2025-06-27 07:23:57 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:23:57 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:23:58 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.175', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:24:01 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:24:03 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.175] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:24:05 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_1_198.18.116.175.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_1_198.18.116.175.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_1_198.18.116.175.json')} 2025-06-27 07:24:05 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_1_198.18.116.175.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_1_198.18.116.175.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_1_198.18.116.175.json')} 2025-06-27 07:24:05 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.116.176 2025-06-27 07:24:05 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.176,nat_gateway=None,authgate=None 2025-06-27 07:24:05 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.176] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:24:05 INFO python_alsi.connection.connection:168 - [198.18.116.176] Connecting as user root 2025-06-27 07:24:05 DEBUG python_alsi.connection.connection:193 - [198.18.116.176] Connecting directly using ssh 2025-06-27 07:24:05 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.176'] 2025-06-27 07:24:09 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.176 2025-06-27 07:24:09 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:24:09 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:24:10 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.176', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:24:13 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:24:15 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.176] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:24:17 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_2_198.18.116.176.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_2_198.18.116.176.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_2_198.18.116.176.json')} 2025-06-27 07:24:17 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_2_198.18.116.176.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_2_198.18.116.176.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_2_198.18.116.176.json')} 2025-06-27 07:24:17 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.18.167.214 2025-06-27 07:24:17 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.167.214,nat_gateway=None,authgate=None 2025-06-27 07:24:17 DEBUG python_alsi.machine.run_cmd:270 - [198.18.167.214] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:24:17 INFO python_alsi.connection.connection:168 - [198.18.167.214] Connecting as user root 2025-06-27 07:24:17 DEBUG python_alsi.connection.connection:193 - [198.18.167.214] Connecting directly using ssh 2025-06-27 07:24:17 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.167.214'] 2025-06-27 07:24:21 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.167.214 2025-06-27 07:24:21 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:24:21 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:24:23 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.167.214', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:24:25 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:24:28 DEBUG python_alsi.machine.run_cmd:270 - [198.18.167.214] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:24:29 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:24:29 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:24:29 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=,ip=198.18.116.168 2025-06-27 07:24:29 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.168,nat_gateway=None,authgate=None 2025-06-27 07:24:29 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:24:29 INFO python_alsi.connection.connection:168 - [198.18.116.168] Connecting as user root 2025-06-27 07:24:29 DEBUG python_alsi.connection.connection:193 - [198.18.116.168] Connecting directly using ssh 2025-06-27 07:24:29 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.168'] 2025-06-27 07:24:33 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.168 2025-06-27 07:24:34 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:24:34 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:24:35 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.168', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:24:37 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:24:40 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:24:41 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:24:41 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:24:41 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtOriginMachine for key network=,ip=198.18.116.179 2025-06-27 07:24:41 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.179,nat_gateway=None,authgate=None 2025-06-27 07:24:41 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.179] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:24:41 INFO python_alsi.connection.connection:168 - [198.18.116.179] Connecting as user root 2025-06-27 07:24:41 DEBUG python_alsi.connection.connection:193 - [198.18.116.179] Connecting directly using ssh 2025-06-27 07:24:41 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.179'] 2025-06-27 07:24:45 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.179 2025-06-27 07:24:46 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:24:46 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:24:47 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.179', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:24:50 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:24:52 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.179] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:24:53 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files {'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None} 2025-06-27 07:24:53 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files {'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None} 2025-06-27 07:24:53 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.228.116 2025-06-27 07:24:53 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.228.116,nat_gateway=None,authgate=None 2025-06-27 07:24:53 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.116] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:24:53 INFO python_alsi.connection.connection:168 - [198.18.228.116] Connecting as user root 2025-06-27 07:24:53 DEBUG python_alsi.connection.connection:193 - [198.18.228.116] Connecting directly using ssh 2025-06-27 07:24:53 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.116'] 2025-06-27 07:24:57 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.228.116 2025-06-27 07:24:58 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:24:58 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:24:59 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.116', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:25:02 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:25:04 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.116] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:25:06 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_5_198.18.228.116.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_5_198.18.228.116.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_5_198.18.228.116.json')} 2025-06-27 07:25:06 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_5_198.18.228.116.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_5_198.18.228.116.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_5_198.18.228.116.json')} 2025-06-27 07:25:06 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.19.11.122 2025-06-27 07:25:06 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.11.122,nat_gateway=None,authgate=None 2025-06-27 07:25:06 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.122] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:25:06 INFO python_alsi.connection.connection:168 - [198.19.11.122] Connecting as user root 2025-06-27 07:25:06 DEBUG python_alsi.connection.connection:193 - [198.19.11.122] Connecting directly using ssh 2025-06-27 07:25:06 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.122'] 2025-06-27 07:25:09 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.11.122 2025-06-27 07:25:10 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:25:10 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:25:11 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.122', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:25:14 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:25:16 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.122] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:25:18 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:25:18 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:25:18 INFO python_mdt.system.mdtsystem._inventory_machines:245 - Ignoring machine 198.18.116.169 from inventory 2025-06-27 07:25:18 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.228.117 2025-06-27 07:25:18 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.228.117,nat_gateway=None,authgate=None 2025-06-27 07:25:18 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.117] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:25:18 INFO python_alsi.connection.connection:168 - [198.18.228.117] Connecting as user root 2025-06-27 07:25:18 DEBUG python_alsi.connection.connection:193 - [198.18.228.117] Connecting directly using ssh 2025-06-27 07:25:18 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.117'] 2025-06-27 07:25:22 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.228.117 2025-06-27 07:25:22 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:25:22 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:25:23 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.117', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:25:26 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:25:28 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.117] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:25:30 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_6_198.18.228.117.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_6_198.18.228.117.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_6_198.18.228.117.json')} 2025-06-27 07:25:30 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_6_198.18.228.117.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_6_198.18.228.117.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_6_198.18.228.117.json')} 2025-06-27 07:25:30 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.18.41.7 2025-06-27 07:25:30 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.41.7,nat_gateway=None,authgate=None 2025-06-27 07:25:30 DEBUG python_alsi.machine.run_cmd:270 - [198.18.41.7] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:25:30 INFO python_alsi.connection.connection:168 - [198.18.41.7] Connecting as user root 2025-06-27 07:25:30 DEBUG python_alsi.connection.connection:193 - [198.18.41.7] Connecting directly using ssh 2025-06-27 07:25:30 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.41.7'] 2025-06-27 07:25:34 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.41.7 2025-06-27 07:25:34 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:25:34 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:25:36 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.41.7', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:25:38 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:25:41 DEBUG python_alsi.machine.run_cmd:270 - [198.18.41.7] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:25:42 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:25:42 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:25:42 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=,ip=198.18.116.170 2025-06-27 07:25:42 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.116.170,nat_gateway=None,authgate=None 2025-06-27 07:25:42 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.170] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:25:42 INFO python_alsi.connection.connection:168 - [198.18.116.170] Connecting as user root 2025-06-27 07:25:42 DEBUG python_alsi.connection.connection:193 - [198.18.116.170] Connecting directly using ssh 2025-06-27 07:25:42 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.170'] 2025-06-27 07:25:46 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.116.170 2025-06-27 07:25:47 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:25:47 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:25:48 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.116.170', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:25:50 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:25:53 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.170] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:25:54 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:25:54 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:25:54 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.228.120 2025-06-27 07:25:54 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.228.120,nat_gateway=None,authgate=None 2025-06-27 07:25:54 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.120] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:25:54 INFO python_alsi.connection.connection:168 - [198.18.228.120] Connecting as user root 2025-06-27 07:25:54 DEBUG python_alsi.connection.connection:193 - [198.18.228.120] Connecting directly using ssh 2025-06-27 07:25:54 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.120'] 2025-06-27 07:25:58 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.228.120 2025-06-27 07:25:59 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:25:59 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:26:00 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.228.120', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:26:03 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:26:05 DEBUG python_alsi.machine.run_cmd:270 - [198.18.228.120] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:26:07 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_7_198.18.228.120.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_7_198.18.228.120.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_7_198.18.228.120.json')} 2025-06-27 07:26:07 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.client_7_198.18.228.120.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.debug_7_198.18.228.120.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.quorum_7_198.18.228.120.json')} 2025-06-27 07:26:07 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.19.11.123 2025-06-27 07:26:07 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.11.123,nat_gateway=None,authgate=None 2025-06-27 07:26:07 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.123] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:26:07 INFO python_alsi.connection.connection:168 - [198.19.11.123] Connecting as user root 2025-06-27 07:26:07 DEBUG python_alsi.connection.connection:193 - [198.19.11.123] Connecting directly using ssh 2025-06-27 07:26:07 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.123'] 2025-06-27 07:26:10 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.11.123 2025-06-27 07:26:11 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:26:11 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:26:12 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.11.123', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:26:15 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:26:17 DEBUG python_alsi.machine.run_cmd:270 - [198.19.11.123] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:26:19 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:26:19 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:26:19 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.19.16.114 2025-06-27 07:26:19 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.16.114,nat_gateway=None,authgate=None 2025-06-27 07:26:19 DEBUG python_alsi.machine.run_cmd:270 - [198.19.16.114] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:26:19 INFO python_alsi.connection.connection:168 - [198.19.16.114] Connecting as user root 2025-06-27 07:26:19 DEBUG python_alsi.connection.connection:193 - [198.19.16.114] Connecting directly using ssh 2025-06-27 07:26:19 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.16.114'] 2025-06-27 07:26:23 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.16.114 2025-06-27 07:26:23 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:26:23 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:26:24 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.16.114', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:26:27 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:26:29 DEBUG python_alsi.machine.run_cmd:270 - [198.19.16.114] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:26:31 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:26:31 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:26:31 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.19.16.116 2025-06-27 07:26:31 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.19.16.116,nat_gateway=None,authgate=None 2025-06-27 07:26:31 DEBUG python_alsi.machine.run_cmd:270 - [198.19.16.116] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:26:31 INFO python_alsi.connection.connection:168 - [198.19.16.116] Connecting as user root 2025-06-27 07:26:31 DEBUG python_alsi.connection.connection:193 - [198.19.16.116] Connecting directly using ssh 2025-06-27 07:26:31 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.16.116'] 2025-06-27 07:26:35 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.19.16.116 2025-06-27 07:26:35 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:26:35 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:26:37 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.19.16.116', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:26:40 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:26:42 DEBUG python_alsi.machine.run_cmd:270 - [198.19.16.116] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:26:44 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:26:44 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:26:44 INFO root.mdt_system_inventory_fixture:142 - As requested, 'mdt-test-auto' won't be included in the inventory. 2025-06-27 07:26:44 INFO root.mdt_system_inventory_fixture:142 - As requested, 'mdt-test2' won't be included in the inventory. 2025-06-27 07:26:44 DEBUG python_mdt.system.mdtsystem._system_info_setup:178 - Setting up MDT system: 'mdt-test3' in Network: 'metanet' 2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=,ip=198.19.8.7 2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=,ip=198.19.3.90 2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=,ip=198.19.11.120 2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=,ip=198.19.16.14 2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=,ip=198.18.88.154 2025-06-27 07:26:45 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.88.154,nat_gateway=None,authgate=None 2025-06-27 07:26:45 DEBUG python_alsi.machine.run_cmd:270 - [198.18.88.154] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:26:45 INFO python_alsi.connection.connection:168 - [198.18.88.154] Connecting as user root 2025-06-27 07:26:45 DEBUG python_alsi.connection.connection:193 - [198.18.88.154] Connecting directly using ssh 2025-06-27 07:26:45 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.88.154'] 2025-06-27 07:26:49 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.88.154 2025-06-27 07:26:50 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:26:50 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:26:51 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.88.154', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:26:54 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:26:56 DEBUG python_alsi.machine.run_cmd:270 - [198.18.88.154] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:26:58 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:26:58 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:26:58 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=,ip=198.18.93.176 2025-06-27 07:26:58 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.93.176,nat_gateway=None,authgate=None 2025-06-27 07:26:58 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:26:58 INFO python_alsi.connection.connection:168 - [198.18.93.176] Connecting as user root 2025-06-27 07:26:58 DEBUG python_alsi.connection.connection:193 - [198.18.93.176] Connecting directly using ssh 2025-06-27 07:26:58 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.93.176'] 2025-06-27 07:27:01 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.93.176 2025-06-27 07:27:02 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:27:02 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:27:03 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.93.176', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:27:06 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:27:08 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:27:10 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:27:10 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:27:10 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtServerMachine for key network=,ip=198.18.101.99 2025-06-27 07:27:10 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.101.99,nat_gateway=None,authgate=None 2025-06-27 07:27:10 DEBUG python_alsi.machine.run_cmd:270 - [198.18.101.99] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:27:10 INFO python_alsi.connection.connection:168 - [198.18.101.99] Connecting as user root 2025-06-27 07:27:10 DEBUG python_alsi.connection.connection:193 - [198.18.101.99] Connecting directly using ssh 2025-06-27 07:27:10 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.101.99'] 2025-06-27 07:27:14 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.101.99 2025-06-27 07:27:14 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:27:14 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:27:15 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.101.99', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:27:18 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:27:20 DEBUG python_alsi.machine.run_cmd:270 - [198.18.101.99] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:27:22 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:27:22 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdt.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': None} 2025-06-27 07:27:22 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.102.196 2025-06-27 07:27:22 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.102.196,nat_gateway=None,authgate=None 2025-06-27 07:27:22 DEBUG python_alsi.machine.run_cmd:270 - [198.18.102.196] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:27:22 INFO python_alsi.connection.connection:168 - [198.18.102.196] Connecting as user root 2025-06-27 07:27:22 DEBUG python_alsi.connection.connection:193 - [198.18.102.196] Connecting directly using ssh 2025-06-27 07:27:22 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.102.196'] 2025-06-27 07:27:26 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.102.196 2025-06-27 07:27:26 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:27:26 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:27:28 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.102.196', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:27:30 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:27:33 DEBUG python_alsi.machine.run_cmd:270 - [198.18.102.196] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:27:34 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_1_198.18.102.196.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_1_198.18.102.196.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_1_198.18.102.196.json')} 2025-06-27 07:27:34 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_1_198.18.102.196.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_1_198.18.102.196.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_1_198.18.102.196.json')} 2025-06-27 07:27:34 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.55.186 2025-06-27 07:27:34 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.55.186,nat_gateway=None,authgate=None 2025-06-27 07:27:34 DEBUG python_alsi.machine.run_cmd:270 - [198.18.55.186] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:27:34 INFO python_alsi.connection.connection:168 - [198.18.55.186] Connecting as user root 2025-06-27 07:27:34 DEBUG python_alsi.connection.connection:193 - [198.18.55.186] Connecting directly using ssh 2025-06-27 07:27:34 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.55.186'] 2025-06-27 07:27:38 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.55.186 2025-06-27 07:27:38 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:27:38 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:27:40 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.55.186', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:27:42 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:27:45 DEBUG python_alsi.machine.run_cmd:270 - [198.18.55.186] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:27:46 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_2_198.18.55.186.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_2_198.18.55.186.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_2_198.18.55.186.json')} 2025-06-27 07:27:46 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_2_198.18.55.186.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_2_198.18.55.186.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_2_198.18.55.186.json')} 2025-06-27 07:27:46 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtZooKeeperMachine for key network=,ip=198.18.88.252 2025-06-27 07:27:46 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.88.252,nat_gateway=None,authgate=None 2025-06-27 07:27:46 DEBUG python_alsi.machine.run_cmd:270 - [198.18.88.252] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:27:46 INFO python_alsi.connection.connection:168 - [198.18.88.252] Connecting as user root 2025-06-27 07:27:46 DEBUG python_alsi.connection.connection:193 - [198.18.88.252] Connecting directly using ssh 2025-06-27 07:27:46 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.88.252'] 2025-06-27 07:27:50 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.88.252 2025-06-27 07:27:51 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:27:51 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:27:52 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.88.252', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:27:54 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:27:57 DEBUG python_alsi.machine.run_cmd:270 - [198.18.88.252] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:27:58 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_3_198.18.88.252.json'), 'debug': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_3_198.18.88.252.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_3_198.18.88.252.json')} 2025-06-27 07:27:58 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.client_3_198.18.88.252.json'), 'debug': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.debug_3_198.18.88.252.json'), 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.json'), 'quorum': PosixPath('/a/ssl-rotation-manager/state/fips.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.quorum_3_198.18.88.252.json')} 2025-06-27 07:27:58 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.18.180.138 2025-06-27 07:27:58 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.180.138,nat_gateway=None,authgate=None 2025-06-27 07:27:58 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.138] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:27:58 INFO python_alsi.connection.connection:168 - [198.18.180.138] Connecting as user root 2025-06-27 07:27:58 DEBUG python_alsi.connection.connection:193 - [198.18.180.138] Connecting directly using ssh 2025-06-27 07:27:58 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.138'] 2025-06-27 07:28:02 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.180.138 2025-06-27 07:28:03 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:28:03 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:28:04 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.138', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:28:06 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:28:09 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.138] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:28:10 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:28:10 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:28:10 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.18.180.141 2025-06-27 07:28:10 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.180.141,nat_gateway=None,authgate=None 2025-06-27 07:28:10 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.141] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:28:10 INFO python_alsi.connection.connection:168 - [198.18.180.141] Connecting as user root 2025-06-27 07:28:10 DEBUG python_alsi.connection.connection:193 - [198.18.180.141] Connecting directly using ssh 2025-06-27 07:28:10 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.141'] 2025-06-27 07:28:14 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.180.141 2025-06-27 07:28:15 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:28:15 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:28:16 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.141', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:28:19 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:28:21 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.141] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:28:23 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:28:23 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:28:23 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.18.93.197 2025-06-27 07:28:23 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.93.197,nat_gateway=None,authgate=None 2025-06-27 07:28:23 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.197] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:28:23 INFO python_alsi.connection.connection:168 - [198.18.93.197] Connecting as user root 2025-06-27 07:28:23 DEBUG python_alsi.connection.connection:193 - [198.18.93.197] Connecting directly using ssh 2025-06-27 07:28:23 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.93.197'] 2025-06-27 07:28:26 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.93.197 2025-06-27 07:28:27 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:28:27 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:28:28 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.93.197', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:28:31 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:28:33 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.197] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:28:35 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:28:35 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:28:35 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtProducerMachine for key network=,ip=198.18.180.134 2025-06-27 07:28:35 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.180.134,nat_gateway=None,authgate=None 2025-06-27 07:28:35 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.134] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:28:35 INFO python_alsi.connection.connection:168 - [198.18.180.134] Connecting as user root 2025-06-27 07:28:35 DEBUG python_alsi.connection.connection:193 - [198.18.180.134] Connecting directly using ssh 2025-06-27 07:28:35 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.134'] 2025-06-27 07:28:38 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.180.134 2025-06-27 07:28:39 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:28:39 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:28:40 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.180.134', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:28:43 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:28:45 DEBUG python_alsi.machine.run_cmd:270 - [198.18.180.134] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:28:47 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files { 'client': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/conf/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:28:47 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files { 'client': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.zk_client.mdtzookeeper_test3.sqa2.qa.akamai.com.0.json'), 'debug': None, 'mdtzk_ca': PosixPath('/a/ssl-rotation-manager/state/fips.mdtproducer.mdtzookeeper_secrets_mdtzookeeper_test3.sqa2.qa.akamai.com_v1.mdtzk_ca.0.json'), 'quorum': None} 2025-06-27 07:28:47 DEBUG python_alsi.util.__call__:474 - Creating a new instance of MdtOriginMachine for key network=,ip=198.18.101.100 2025-06-27 07:28:47 DEBUG python_alsi.util.__call__:474 - Creating a new instance of ConnManager for key ip=198.18.101.100,nat_gateway=None,authgate=None 2025-06-27 07:28:47 DEBUG python_alsi.machine.run_cmd:270 - [198.18.101.100] Running 'ls /a/ssl-rotation-manager/conf/*.json' as root 2025-06-27 07:28:47 INFO python_alsi.connection.connection:168 - [198.18.101.100] Connecting as user root 2025-06-27 07:28:47 DEBUG python_alsi.connection.connection:193 - [198.18.101.100] Connecting directly using ssh 2025-06-27 07:28:47 DEBUG python_alsi.connection.initialize_pipe:665 - Connection command: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.101.100'] 2025-06-27 07:28:51 DEBUG python_alsi.connection.initialize_pipe:669 - Established contact with 198.18.101.100 2025-06-27 07:28:51 DEBUG python_alsi.connection._setup_rpyc:485 - Waiting for remote shell. 2025-06-27 07:28:51 DEBUG python_alsi.connection._setup_rpyc:492 - Starting up rpyc on the remote machine (Python: /usr/local/akamai/bin/python3.11). 2025-06-27 07:28:52 DEBUG python_alsi.connection.__init__:685 - Connecting to remote rpyc server with: ['ssh', '-e', 'none', '-A', '-oControlMaster=auto', '-oControlPersist=10m', '-oControlPath=/home/rjagarla/.ssh/controlmasters/None-%r@%h:%p', '-oStrictHostKeyChecking=no', '-oBatchMode=yes', '-l', 'root', '198.18.101.100', 'sh', '-c', '"echo SYNC0;exec sh"'] 2025-06-27 07:28:55 DEBUG python_alsi.connection.__init__:696 - Connected. 2025-06-27 07:28:57 DEBUG python_alsi.machine.run_cmd:270 - [198.18.101.100] Running 'ls /a/ssl-rotation-manager/state/*.json' as root 2025-06-27 07:28:59 INFO python_mdt.service.srm._map_config_paths:78 - SRM config files {'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None} 2025-06-27 07:28:59 INFO python_mdt.service.srm._map_config_paths:79 - SRM state files {'client': None, 'debug': None, 'mdtzk_ca': None, 'quorum': None} 2025-06-27 07:28:59 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=,ip=198.18.101.101 2025-06-27 07:28:59 DEBUG python_alsi.util.__call__:480 - Returning existing instance of NetworkMachine for key network=,ip=198.18.101.102 2025-06-27 07:28:59 INFO python_mdt.system.mdtsystem._inventory_machines:245 - Ignoring machine 198.18.116.169 from inventory 2025-06-27 07:28:59 INFO root.mdt_channel_injector_fixture:489 - Injecting MDT channels into the MDT system. 2025-06-27 07:28:59 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtZooKeeperHelper - ['dynamic_quorum_config_data', 'servers_post_subscription_config_data', 'servers_pre_subscription_config_data'] 2025-06-27 07:28:59 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.116.175, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:29:04 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtzookeeper.quorum.freeflow' MDT system. 2025-06-27 07:29:04 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.168, network=metanet, testnet=sqa2, role=server 2025-06-27 07:29:10 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtzookeeper.quorum.freeflow'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:29:15 DEBUG python_alsi.service.get_local_tableprov_data:414 - Found matching row for machine 198.18.116.168 in table mdtreceiver_status: {'machineip': '198.18.116.168', 'version': '2.22', 'reporting_time': '1751009322', 'client_connection_limit': '110', 'client_connections': '34', 'max_client_connections_1h': '35', 'heap_used': '338684256', 'heap_committed': '1090519040', 'heap_max': '12884901888', 'nonheap_used': '105102728', 'nonheap_committed': '138674176', 'bad_zk_state_1h': '0', 'mdtzookeeper_quorum': 'mdtzookeeper.sqa2.qa.akamai.com', 'zk_op_successes_1m': '9', 'zk_op_failures_1m': '0', 'zk_op_errors_1m': '0', 'zk_op_successes_5m': '122', 'zk_op_failures_5m': '0', 'zk_op_errors_5m': '0', 'zk_op_successes_15m': '320', 'zk_op_failures_15m': '0', 'zk_op_errors_15m': '0'} 2025-06-27 07:29:16 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'source /usr/local/akamai/etc/akamai.conf; $MDT_JAVA_HOME/bin/keytool -importkeystore -srckeystore /usr/local/akamai/mdtreceiver/etc/keys/api.jks -srcstorepass password -destkeystore /tmp/python-mdt-secrets/api.p12.tmp -deststorepass password -deststoretype pkcs12 -noprompt -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /a/mdtreceiver/lib/bc-fips.jar -srcstoretype BCFKS' as root 2025-06-27 07:29:19 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'openssl pkcs12 -in /tmp/python-mdt-secrets/api.p12 -nodes -nocerts -out /tmp/python-mdt-secrets/api.private_key.tmp -passin pass:password' as root 2025-06-27 07:29:20 DEBUG python_alsi.machine.run_cmd:270 - [198.18.116.168] Running 'openssl pkcs12 -in /tmp/python-mdt-secrets/api.p12 -nodes -nokeys -out /tmp/python-mdt-secrets/api.certificate.tmp -passin pass:password' as root 2025-06-27 07:29:23 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtzookeeper_servers.postinstall.csv' MDT system. 2025-06-27 07:29:24 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:29:24 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtzookeeper_servers.preinstall.csv' MDT system. 2025-06-27 07:29:26 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:29:26 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'dynamic_quorum_config_data': , 'servers_post_subscription_config_data': , 'servers_pre_subscription_config_data': } 2025-06-27 07:29:26 INFO root.inject_channels:466 - MDTZK ip: 198.18.116.175 2025-06-27 07:29:26 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=) 2025-06-27 07:29:26 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:29:26 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:29:26 INFO root.inject_channels:466 - MDTZK ip: 198.18.228.117 2025-06-27 07:29:26 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.228.117, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:29:31 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=) 2025-06-27 07:29:31 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:29:31 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:29:31 INFO root.inject_channels:466 - MDTZK ip: 198.18.116.177 2025-06-27 07:29:31 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.116.177, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:29:36 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=) 2025-06-27 07:29:36 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:29:36 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:29:36 INFO root.inject_channels:466 - MDTZK ip: 198.18.228.115 2025-06-27 07:29:36 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.228.115, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:29:41 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=) 2025-06-27 07:29:41 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:29:41 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:29:41 INFO root.inject_channels:466 - MDTZK ip: 198.18.228.120 2025-06-27 07:29:41 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.228.120, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:29:46 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=) 2025-06-27 07:29:46 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:29:46 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:29:46 INFO root.inject_channels:466 - MDTZK ip: 198.18.228.116 2025-06-27 07:29:46 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.228.116, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:29:51 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=) 2025-06-27 07:29:51 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:29:51 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:29:51 INFO root.inject_channels:466 - MDTZK ip: 198.18.116.176 2025-06-27 07:29:51 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.116.176, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:29:56 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.freeflow', port=5874, _mdt_channel=) 2025-06-27 07:29:56 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:29:56 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:29:56 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtProducer - ['ensemble_channel_config_data', 'servers_post_channel_config_data', 'servers_pre_channel_config_data'] 2025-06-27 07:29:56 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.11.122, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:30:01 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtproducer.sets.producer_ensemble1' MDT system. 2025-06-27 07:30:02 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtproducer.sets.producer_ensemble1'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:30:02 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtproducer_servers.postinstall.csv' MDT system. 2025-06-27 07:30:04 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtproducer_servers.postinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:30:04 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtproducer_servers.preinstall.csv' MDT system. 2025-06-27 07:30:05 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtproducer_servers.preinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:30:05 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'ensemble_channel_config_data': , 'servers_post_channel_config_data': , 'servers_pre_channel_config_data': } 2025-06-27 07:30:05 INFO root.inject_channels:476 - MDTProducer ip: 198.19.11.122 2025-06-27 07:30:05 INFO root.inject_channels:476 - MDTProducer ip: 198.18.167.214 2025-06-27 07:30:05 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.167.214, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:30:10 INFO root.inject_channels:476 - MDTProducer ip: 198.18.41.7 2025-06-27 07:30:10 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.41.7, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:30:15 INFO root.inject_channels:476 - MDTProducer ip: 198.19.11.121 2025-06-27 07:30:15 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.11.121, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:30:20 INFO root.inject_channels:476 - MDTProducer ip: 198.19.11.123 2025-06-27 07:30:20 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.11.123, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:30:25 INFO root.inject_channels:476 - MDTProducer ip: 198.19.16.116 2025-06-27 07:30:25 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.16.116, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:30:30 INFO root.inject_channels:476 - MDTProducer ip: 198.19.16.114 2025-06-27 07:30:30 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.19.16.114, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:30:35 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtReceiver - ['dynamic_channel_config_data', 'group_channel_config_data'] 2025-06-27 07:30:35 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtreceiver.freeflow' MDT system. 2025-06-27 07:30:36 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtreceiver.freeflow'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:30:36 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.groups.freeflow' MDT system. 2025-06-27 07:30:38 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.groups.freeflow'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:30:38 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'dynamic_channel_config_data': , 'group_channel_config_data': } 2025-06-27 07:30:38 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.168 2025-06-27 07:30:38 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.173 2025-06-27 07:30:38 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.173, network=metanet, testnet=sqa2, role=server 2025-06-27 07:30:43 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.171 2025-06-27 07:30:43 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.171, network=metanet, testnet=sqa2, role=server 2025-06-27 07:30:48 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.172 2025-06-27 07:30:48 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.172, network=metanet, testnet=sqa2, role=server 2025-06-27 07:30:53 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.170 2025-06-27 07:30:53 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.170, network=metanet, testnet=sqa2, role=server 2025-06-27 07:30:58 INFO root.inject_channels:484 - MDTServer ip: 198.18.116.174 2025-06-27 07:30:58 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.116.174, network=metanet, testnet=sqa2, role=server 2025-06-27 07:31:03 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtZooKeeperHelper - ['dynamic_quorum_config_data', 'servers_post_subscription_config_data', 'servers_pre_subscription_config_data'] 2025-06-27 07:31:03 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.55.186, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:31:08 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtzookeeper.quorum.mdt-test3' MDT system. 2025-06-27 07:31:09 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtzookeeper.quorum.mdt-test3'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:31:09 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtzookeeper_servers.postinstall.csv' MDT system. 2025-06-27 07:31:11 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:31:11 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtzookeeper_servers.preinstall.csv' MDT system. 2025-06-27 07:31:12 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:31:12 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'dynamic_quorum_config_data': , 'servers_post_subscription_config_data': , 'servers_pre_subscription_config_data': } 2025-06-27 07:31:12 INFO root.inject_channels:466 - MDTZK ip: 198.18.55.186 2025-06-27 07:31:12 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.mdt-test3', port=5874, _mdt_channel=) 2025-06-27 07:31:12 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:31:12 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:31:12 INFO root.inject_channels:466 - MDTZK ip: 198.18.88.252 2025-06-27 07:31:12 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.88.252, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:31:17 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.mdt-test3', port=5874, _mdt_channel=) 2025-06-27 07:31:17 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:31:17 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:31:17 INFO root.inject_channels:466 - MDTZK ip: 198.18.102.196 2025-06-27 07:31:17 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtzookeeper_helper configuration from service=mdtzookeeper_helper, MdtZooKeeperMachine: ip=198.18.102.196, network=metanet, testnet=sqa2, role=zookeeper 2025-06-27 07:31:22 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/mdt.config.mdtzookeeper.quorum'), notify_token='mdt.config.mdtzookeeper.quorum.mdt-test3', port=5874, _mdt_channel=) 2025-06-27 07:31:22 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.postinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.postinstall.csv', port=5876, _mdt_channel=) 2025-06-27 07:31:22 INFO root.inject_channels:470 - MDT ZooKeeper channels: ChannelConfigData(destination=PosixPath('/usr/local/akamai/mdtzookeeper/etc/staticinfo.metanet.mdtzookeeper_servers.preinstall.csv'), notify_token='staticinfo.metanet.mdtzookeeper_servers.preinstall.csv', port=5875, _mdt_channel=) 2025-06-27 07:31:22 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtProducer - ['ensemble_channel_config_data', 'servers_post_channel_config_data', 'servers_pre_channel_config_data'] 2025-06-27 07:31:22 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.180.141, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:31:27 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtproducer.sets.producer_ensemble_test3' MDT system. 2025-06-27 07:31:29 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtproducer.sets.producer_ensemble_test3'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:31:29 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtproducer_servers.postinstall.csv' MDT system. 2025-06-27 07:31:30 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtproducer_servers.postinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:31:30 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'staticinfo.metanet.mdtproducer_servers.preinstall.csv' MDT system. 2025-06-27 07:31:31 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'staticinfo.metanet.mdtproducer_servers.preinstall.csv'. Namespace match: '{'group': 'aggconf', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:31:31 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'ensemble_channel_config_data': , 'servers_post_channel_config_data': , 'servers_pre_channel_config_data': } 2025-06-27 07:31:31 INFO root.inject_channels:476 - MDTProducer ip: 198.18.180.141 2025-06-27 07:31:31 INFO root.inject_channels:476 - MDTProducer ip: 198.18.180.138 2025-06-27 07:31:31 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.180.138, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:31:36 INFO root.inject_channels:476 - MDTProducer ip: 198.18.180.134 2025-06-27 07:31:36 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.180.134, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:31:42 INFO root.inject_channels:476 - MDTProducer ip: 198.18.93.197 2025-06-27 07:31:42 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtproducer configuration from service=mdtproducer, MdtProducerMachine: ip=198.18.93.197, network=metanet, testnet=sqa2, role=producer 2025-06-27 07:31:47 INFO root._get_injectable_func_names:436 - Injectable functions found in class name: MdtReceiver - ['dynamic_channel_config_data', 'group_channel_config_data'] 2025-06-27 07:31:47 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:31:52 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.mdtreceiver.mdt-test3' MDT system. 2025-06-27 07:31:53 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.mdtreceiver.mdt-test3'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:31:53 INFO python_mdt.channel.mdtchannel.find_channel:365 - Searching for 'freeflow' in 'mdt.config.groups.mdt-test3' MDT system. 2025-06-27 07:31:55 INFO python_mdt.channel.mdtchannel.find_channel:372 - Found requested channel name: 'mdt.config.groups.mdt-test3'. Namespace match: '{'group': 'mdt_config', 'type': 'FILE'}', submitter client of MDT system: 'freeflow' 2025-06-27 07:31:55 INFO root._map_func_to_mdt_channel:452 - Injectable channel map: {'dynamic_channel_config_data': , 'group_channel_config_data': } 2025-06-27 07:31:55 INFO root.inject_channels:484 - MDTServer ip: 198.18.93.176 2025-06-27 07:31:55 INFO root.inject_channels:484 - MDTServer ip: 198.18.101.99 2025-06-27 07:31:55 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.101.99, network=metanet, testnet=sqa2, role=server 2025-06-27 07:32:00 INFO root.inject_channels:484 - MDTServer ip: 198.18.88.154 2025-06-27 07:32:00 DEBUG python_mdt.service.mdtservice.get_config:92 - Fetching current mdtreceiver configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.88.154, network=metanet, testnet=sqa2, role=server 2025-06-27 07:32:05 DEBUG root.mdtserver_machines_fixture:221 - Initializing mdtserver_machines fixture 2025-06-27 07:32:05 DEBUG root.mdtproducer_machines_fixture:239 - Initializing mdtproducer_machines fixture 2025-06-27 07:32:05 INFO root.verify_testnet_fixture:323 - Skipping testnet validation 2025-06-27 07:32:05 INFO conftest.fixture_mdtserver_machine:32 - Selected 198.18.93.176 as the mdtreceiver under test. 2025-06-27 07:32:12 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'ls /root/backup/usr/local/akamai/mdtreceiver/etc/mdt.config.groups*.bak' as root 2025-06-27 07:32:13 WARNING python_alsi.machine.backup_file:509 - WARNING - previously backed up version of /usr/local/akamai/mdtreceiver/etc/mdt.config.groups found in /root/backup/usr/local/akamai/mdtreceiver/etc 2025-06-27 07:32:18 DEBUG python_alsi.machine.backup_file:521 - Backing up file /usr/local/akamai/mdtreceiver/etc/mdt.config.groups to /root/backup/usr/local/akamai/mdtreceiver/etc/mdt.config.groups.2xyC.bak on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:32:29 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'ls /root/backup/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver*.bak' as root 2025-06-27 07:32:31 WARNING python_alsi.machine.backup_file:509 - WARNING - previously backed up version of /usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver found in /root/backup/usr/local/akamai/mdtreceiver/etc 2025-06-27 07:32:36 DEBUG python_alsi.machine.backup_file:521 - Backing up file /usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver to /root/backup/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver.1POS.bak on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server -------------------------------------------------------------------------------------------------- live log call -------------------------------------------------------------------------------------------------- 2025-06-27 07:32:39 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_default_limits_are_set_in_mdtreceiver_config' ==================== 2025-06-27 07:32:39 INFO test_channel_limits_new.test_default_limits_are_set_in_mdtreceiver_config:178 - Checking the default channel config settings on MdtReceiver 198.18.93.176. 2025-06-27 07:32:39 INFO root.pytest_runtest_makereport:52 - Test 'test_default_limits_are_set_in_mdtreceiver_config' passed PASSED [ 16%] ------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------ 2025-06-27 07:32:39 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_default_limits_are_set_in_mdtreceiver_config -> METANET-2537 tests/mdtreceiver/test_channel_limits_new.py::test_limits_from_granular_section_are_enforced_for_file_channel ------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------- 2025-06-27 07:32:39 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server -------------------------------------------------------------------------------------------------- live log call -------------------------------------------------------------------------------------------------- 2025-06-27 07:32:43 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_from_granular_section_are_enforced_for_file_channel' ==================== 2025-06-27 07:32:43 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_1.*' 2025-06-27 07:32:43 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_1.*' in 'mdt-test3' MDT system. 2025-06-27 07:32:45 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_1.* 2025-06-27 07:32:49 DEBUG python_alsi.service.get_local_tableprov_data:414 - Found matching row for machine 198.18.93.176 in table mdtreceiver_status: {'machineip': '198.18.93.176', 'version': '2.22', 'reporting_time': '1751009545', 'client_connection_limit': '110', 'client_connections': '31', 'max_client_connections_1h': '31', 'heap_used': '74435864', 'heap_committed': '150994944', 'heap_max': '12884901888', 'nonheap_used': '97399752', 'nonheap_committed': '110886912', 'bad_zk_state_1h': '0', 'mdtzookeeper_quorum': 'mdtzookeeper_test3.sqa2.qa.akamai.com', 'zk_op_successes_1m': '0', 'zk_op_failures_1m': '0', 'zk_op_errors_1m': '0', 'zk_op_successes_5m': '0', 'zk_op_failures_5m': '0', 'zk_op_errors_5m': '0', 'zk_op_successes_15m': '0', 'zk_op_failures_15m': '0', 'zk_op_errors_15m': '0'} 2025-06-27 07:32:49 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'source /usr/local/akamai/etc/akamai.conf; $MDT_JAVA_HOME/bin/keytool -importkeystore -srckeystore /usr/local/akamai/mdtreceiver/etc/keys/api.jks -srcstorepass password -destkeystore /tmp/python-mdt-secrets/api.p12.tmp -deststorepass password -deststoretype pkcs12 -noprompt -provider org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -providerpath /a/mdtreceiver/lib/bc-fips.jar -srcstoretype BCFKS' as root 2025-06-27 07:32:53 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'openssl pkcs12 -in /tmp/python-mdt-secrets/api.p12 -nodes -nocerts -out /tmp/python-mdt-secrets/api.private_key.tmp -passin pass:password' as root 2025-06-27 07:32:55 DEBUG python_alsi.machine.run_cmd:270 - [198.18.93.176] Running 'openssl pkcs12 -in /tmp/python-mdt-secrets/api.p12 -nodes -nokeys -out /tmp/python-mdt-secrets/api.certificate.tmp -passin pass:password' as root 2025-06-27 07:32:57 INFO test_channel_limits_new.test_limits_from_granular_section_are_enforced_for_file_channel:204 - Setting 'max_channel_mb' to 2 MB for channel test_mdt-test3.multi_1.W4Ll. 2025-06-27 07:32:57 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:33:02 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:33:05 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:33:10 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:33:12 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009588.mdti2q-g 2025-06-27 07:33:17 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:33:19 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009588.mdti2q-g 2025-06-27 07:33:25 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:33:26 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009589 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:33:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009589 2025-06-27 07:33:41 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1750932290, "numBadAcks": 0, "mdtTimestamp": 1750932222, "mdtTimestampMillisPart": 0} 2025-06-27 07:33:41 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009589 or higher (current=1750932222); sleeping 5s (600s remaining) 2025-06-27 07:33:57 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0} 2025-06-27 07:33:57 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009589 2025-06-27 07:33:57 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009589 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:33:57 INFO test_channel_limits_new._prepare_content:45 - Preparing channel test_mdt-test3.multi_1.W4Ll with file of size 3145728 bytes. 2025-06-27 07:33:57 INFO python_mdt.channel.content.generate_random_file_content:163 - Generating a 3145728 byte random text file 2025-06-27 07:33:57 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_1.W4Ll with 1 files of size 3145728 bytes. 2025-06-27 07:34:00 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_1.W4Ll?zone=1&submitType=full 2025-06-27 07:34:07 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_1.W4Ll 2025-06-27 07:34:08 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_1.W4Ll.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded channel size limit","status":413,"detail":"Submission size exceeds the allowed limit of 2097152 bytes","requestID":"REQID-a9907b0e-3f0e-4d55-a8f4-17c7631e3c89-mdt-test-8"}') 2025-06-27 07:34:08 INFO test_channel_limits_new._prepare_content:45 - Preparing channel test_mdt-test3.multi_1.W4Ll with file of size 1048576 bytes. 2025-06-27 07:34:08 INFO python_mdt.channel.content.generate_random_file_content:163 - Generating a 1048576 byte random text file 2025-06-27 07:34:08 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_1.W4Ll with 1 files of size 1048576 bytes. 2025-06-27 07:34:09 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_1.W4Ll?zone=1&submitType=full 2025-06-27 07:34:14 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_1.W4Ll 2025-06-27 07:34:15 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_1.W4Ll with in channel limits is succeeded. 2025-06-27 07:34:15 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_from_granular_section_are_enforced_for_file_channel' passed PASSED [ 33%] ------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------ 2025-06-27 07:34:15 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_from_granular_section_are_enforced_for_file_channel -> METANET-2543 2025-06-27 07:34:15 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation 2025-06-27 07:34:15 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:34:17 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:34:22 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:34:24 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009660.mdti2q-g 2025-06-27 07:34:29 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:34:31 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009660.mdti2q-g 2025-06-27 07:34:37 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:34:38 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009660 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:34:38 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009660 2025-06-27 07:34:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0} 2025-06-27 07:34:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (600s remaining) 2025-06-27 07:35:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0} 2025-06-27 07:35:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (595s remaining) 2025-06-27 07:35:21 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0} 2025-06-27 07:35:21 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (590s remaining) 2025-06-27 07:35:37 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0} 2025-06-27 07:35:37 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (585s remaining) 2025-06-27 07:35:53 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3bc5540f77abeb9be5d964f1023c82c8e375932ef0ab0c8a319d9e00cfe7ba8a", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009628, "numBadAcks": 0, "mdtTimestamp": 1751009589, "mdtTimestampMillisPart": 0} 2025-06-27 07:35:53 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009660 or higher (current=1751009589); sleeping 5s (580s remaining) 2025-06-27 07:36:09 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009753, "numBadAcks": 0, "mdtTimestamp": 1751009660, "mdtTimestampMillisPart": 0} 2025-06-27 07:36:09 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009660 2025-06-27 07:36:09 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009660 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server tests/mdtreceiver/test_channel_limits_new.py::test_limits_for_ns_are_enforced_for_dir_channel ------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------- 2025-06-27 07:36:09 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server -------------------------------------------------------------------------------------------------- live log call -------------------------------------------------------------------------------------------------- 2025-06-27 07:36:13 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_for_ns_are_enforced_for_dir_channel' ==================== 2025-06-27 07:36:13 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*' 2025-06-27 07:36:13 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system. 2025-06-27 07:36:15 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.* 2025-06-27 07:36:15 INFO test_channel_limits_new.test_limits_for_ns_are_enforced_for_dir_channel:226 - Setting limits {'max_channel_file_count': 10, 'max_channel_mb': 20} for namespace test_mdt-test3.multi_2.dir.*. 2025-06-27 07:36:15 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:36:19 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:36:21 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:36:26 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:36:28 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009784.mdti2q-g 2025-06-27 07:36:33 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:36:35 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009784.mdti2q-g 2025-06-27 07:36:40 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:36:40 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009784 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:36:40 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009784 2025-06-27 07:36:51 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009753, "numBadAcks": 0, "mdtTimestamp": 1751009660, "mdtTimestampMillisPart": 0} 2025-06-27 07:36:51 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009784 or higher (current=1751009660); sleeping 5s (600s remaining) 2025-06-27 07:37:07 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009753, "numBadAcks": 0, "mdtTimestamp": 1751009660, "mdtTimestampMillisPart": 0} 2025-06-27 07:37:07 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009784 or higher (current=1751009660); sleeping 5s (595s remaining) 2025-06-27 07:37:23 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009753, "numBadAcks": 0, "mdtTimestamp": 1751009660, "mdtTimestampMillisPart": 0} 2025-06-27 07:37:23 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009784 or higher (current=1751009660); sleeping 5s (590s remaining) 2025-06-27 07:37:39 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3c20f2dd1fcf746e1c690b60a183d2aa4ef4d12907dbe2025dea59617a5a7365", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009856, "numBadAcks": 0, "mdtTimestamp": 1751009784, "mdtTimestampMillisPart": 0} 2025-06-27 07:37:39 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009784 2025-06-27 07:37:39 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009784 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:37:39 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.0vWN with 15 files of size 1048576 bytes. 2025-06-27 07:37:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:37:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file 2025-06-27 07:37:40 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.0vWN with 15 files of size 1048576 bytes. 2025-06-27 07:37:42 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN?zone=1&submitType=full 2025-06-27 07:37:51 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN 2025-06-27 07:37:52 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.0vWN.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 15, exceeding the configured limit of 10","requestID":"REQID-116d6386-e530-4651-b97f-84fab09ffbb4-mdt-test-8"}') 2025-06-27 07:37:52 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.0vWN with 5 files of size 5242880 bytes. 2025-06-27 07:37:52 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 5242880 byte random text file 2025-06-27 07:37:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 5242880 byte random text file 2025-06-27 07:37:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 5242880 byte random text file 2025-06-27 07:37:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 5242880 byte random text file 2025-06-27 07:37:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 5242880 byte random text file 2025-06-27 07:37:54 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.0vWN with 5 files of size 5242880 bytes. 2025-06-27 07:37:55 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN?zone=1&submitType=full 2025-06-27 07:38:06 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN 2025-06-27 07:38:06 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.0vWN.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded channel size limit","status":413,"detail":"Submission size exceeds the allowed limit of 20971520 bytes","requestID":"REQID-bc23c06e-968e-4355-8ebe-238d19bee25f-mdt-test-8"}') 2025-06-27 07:38:06 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.0vWN with 10 files of size 1048576 bytes. 2025-06-27 07:38:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:38:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file 2025-06-27 07:38:07 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file 2025-06-27 07:38:07 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.0vWN with 10 files of size 1048576 bytes. 2025-06-27 07:38:09 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN?zone=1&submitType=full 2025-06-27 07:38:17 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.0vWN 2025-06-27 07:38:17 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.0vWN with in channel limits is succeeded. 2025-06-27 07:38:17 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_for_ns_are_enforced_for_dir_channel' passed PASSED [ 50%] ------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------ 2025-06-27 07:38:17 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_for_ns_are_enforced_for_dir_channel -> METANET-2538 2025-06-27 07:38:17 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation 2025-06-27 07:38:17 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:38:19 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:38:24 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:38:26 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009902.mdti2q-g 2025-06-27 07:38:31 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:38:33 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009902.mdti2q-g 2025-06-27 07:38:37 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:38:38 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009902 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:38:38 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009902 2025-06-27 07:38:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3c20f2dd1fcf746e1c690b60a183d2aa4ef4d12907dbe2025dea59617a5a7365", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009856, "numBadAcks": 0, "mdtTimestamp": 1751009784, "mdtTimestampMillisPart": 0} 2025-06-27 07:38:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009902 or higher (current=1751009784); sleeping 5s (600s remaining) 2025-06-27 07:39:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3c20f2dd1fcf746e1c690b60a183d2aa4ef4d12907dbe2025dea59617a5a7365", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009856, "numBadAcks": 0, "mdtTimestamp": 1751009784, "mdtTimestampMillisPart": 0} 2025-06-27 07:39:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009902 or higher (current=1751009784); sleeping 5s (595s remaining) 2025-06-27 07:39:21 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "3c20f2dd1fcf746e1c690b60a183d2aa4ef4d12907dbe2025dea59617a5a7365", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009856, "numBadAcks": 0, "mdtTimestamp": 1751009784, "mdtTimestampMillisPart": 0} 2025-06-27 07:39:21 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751009902 or higher (current=1751009784); sleeping 5s (590s remaining) 2025-06-27 07:39:37 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751009969, "numBadAcks": 0, "mdtTimestamp": 1751009902, "mdtTimestampMillisPart": 0} 2025-06-27 07:39:37 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009902 2025-06-27 07:39:37 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009902 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server tests/mdtreceiver/test_channel_limits_new.py::test_limits_are_enforced_on_resulting_channel_state ------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------- 2025-06-27 07:39:37 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server -------------------------------------------------------------------------------------------------- live log call -------------------------------------------------------------------------------------------------- 2025-06-27 07:39:41 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_are_enforced_on_resulting_channel_state' ==================== 2025-06-27 07:39:41 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*' 2025-06-27 07:39:41 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system. 2025-06-27 07:39:43 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.* 2025-06-27 07:39:43 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:248 - Setting limits {'max_channel_file_count': 10, 'max_channel_mb': 20} for namespace test_mdt-test3.multi_2.dir.gZGP. 2025-06-27 07:39:43 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:39:47 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:39:49 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:39:54 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:39:56 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751009992.mdti2q-g 2025-06-27 07:40:00 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:40:02 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751009992.mdti2q-g 2025-06-27 07:40:07 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:40:07 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751009992 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:40:07 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751009992 2025-06-27 07:40:18 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "22baebe4e376045da6704aa211abdaf9555620a20f65632e0b001408176d9db4", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010012, "numBadAcks": 0, "mdtTimestamp": 1751009992, "mdtTimestampMillisPart": 0} 2025-06-27 07:40:18 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751009992 2025-06-27 07:40:18 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751009992 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:40:18 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.gZGP with 5 files of size 1048576 bytes. 2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:40:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:40:18 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.gZGP with 5 files of size 1048576 bytes. 2025-06-27 07:40:20 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP?zone=1&submitType=full 2025-06-27 07:40:27 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP 2025-06-27 07:40:27 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.gZGP with in channel limits is succeeded. 2025-06-27 07:40:27 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:257 - Submitting content such that the resulting channel state exceeds the channel size limit. 2025-06-27 07:40:27 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.gZGP with 4 files of size 4194304 bytes. 2025-06-27 07:40:28 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 4194304 byte random text file 2025-06-27 07:40:28 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 4194304 byte random text file 2025-06-27 07:40:28 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 4194304 byte random text file 2025-06-27 07:40:28 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 4194304 byte random text file 2025-06-27 07:40:30 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP?zone=1&submitType=incr_lenient 2025-06-27 07:40:39 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP 2025-06-27 07:40:39 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:266 - Incremental lenient submission for channel test_mdt-test3.multi_2.dir.gZGP.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded channel size limit","status":413,"detail":"Operation will result in a channel size of 22020096 bytes, exceeding the configured limit of 20971520 bytes","requestID":"REQID-e290595d-35a1-4576-9aee-01371271375d-mdt-test-8"}' 2025-06-27 07:40:39 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:257 - Submitting content such that the resulting channel state exceeds the file count limit. 2025-06-27 07:40:39 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.gZGP with 6 files of size 1048576 bytes. 2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file 2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file 2025-06-27 07:40:39 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file 2025-06-27 07:40:41 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP?zone=1&submitType=incr_lenient 2025-06-27 07:40:47 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.gZGP 2025-06-27 07:40:47 INFO test_channel_limits_new.test_limits_are_enforced_on_resulting_channel_state:266 - Incremental lenient submission for channel test_mdt-test3.multi_2.dir.gZGP.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 11, exceeding the configured limit of 10","requestID":"REQID-4477923d-3098-4062-84aa-20a4ff922e7b-mdt-test-8"}' 2025-06-27 07:40:47 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_are_enforced_on_resulting_channel_state' passed PASSED [ 66%] ------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------ 2025-06-27 07:40:47 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_are_enforced_on_resulting_channel_state -> METANET-2540 2025-06-27 07:40:47 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation 2025-06-27 07:40:47 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:40:49 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:40:53 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:40:55 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010052.mdti2q-g 2025-06-27 07:41:00 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:41:02 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010052.mdti2q-g 2025-06-27 07:41:06 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:41:07 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010052 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:41:07 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010052 2025-06-27 07:41:17 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "22baebe4e376045da6704aa211abdaf9555620a20f65632e0b001408176d9db4", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010012, "numBadAcks": 0, "mdtTimestamp": 1751009992, "mdtTimestampMillisPart": 0} 2025-06-27 07:41:17 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010052 or higher (current=1751009992); sleeping 5s (600s remaining) 2025-06-27 07:41:33 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "22baebe4e376045da6704aa211abdaf9555620a20f65632e0b001408176d9db4", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010012, "numBadAcks": 0, "mdtTimestamp": 1751009992, "mdtTimestampMillisPart": 0} 2025-06-27 07:41:33 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010052 or higher (current=1751009992); sleeping 5s (595s remaining) 2025-06-27 07:41:48 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010093, "numBadAcks": 0, "mdtTimestamp": 1751010052, "mdtTimestampMillisPart": 0} 2025-06-27 07:41:48 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010052 2025-06-27 07:41:48 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010052 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server tests/mdtreceiver/test_channel_limits_new.py::test_limits_can_be_set_at_subnamespace ------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------- 2025-06-27 07:41:48 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server -------------------------------------------------------------------------------------------------- live log call -------------------------------------------------------------------------------------------------- 2025-06-27 07:41:53 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_can_be_set_at_subnamespace' ==================== 2025-06-27 07:41:53 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:41:57 INFO test_channel_limits_new._test_runner:131 - Setting channel limits for multiple channels/namespaces. 2025-06-27 07:41:57 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*' 2025-06-27 07:41:57 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system. 2025-06-27 07:41:58 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.* 2025-06-27 07:41:58 INFO test_channel_limits_new._test_runner:135 - Test channel is test_mdt-test3.multi_2.dir.postfix1.channel1. 2025-06-27 07:41:58 INFO test_channel_limits_new._helper_to_update_granular_config:113 - Setting limits {'max_channel_file_count': 10, 'max_channel_mb': 20} for test_mdt-test3.multi_2.dir.postfix1.*. 2025-06-27 07:41:58 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*' 2025-06-27 07:41:58 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system. 2025-06-27 07:42:00 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.* 2025-06-27 07:42:00 INFO test_channel_limits_new._test_runner:135 - Test channel is test_mdt-test3.multi_2.dir.postfix2.channel2. 2025-06-27 07:42:00 INFO test_channel_limits_new._helper_to_update_granular_config:113 - Setting limits {'max_channel_file_count': 20, 'max_channel_mb': 30} for test_mdt-test3.multi_2.dir.postfix2.*. 2025-06-27 07:42:00 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:42:01 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:42:06 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:42:08 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010124.mdti2q-g 2025-06-27 07:42:12 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:42:14 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010124.mdti2q-g 2025-06-27 07:42:19 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:42:19 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010125 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:42:19 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010125 2025-06-27 07:42:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010093, "numBadAcks": 0, "mdtTimestamp": 1751010052, "mdtTimestampMillisPart": 0} 2025-06-27 07:42:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010125 or higher (current=1751010052); sleeping 5s (600s remaining) 2025-06-27 07:42:45 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010093, "numBadAcks": 0, "mdtTimestamp": 1751010052, "mdtTimestampMillisPart": 0} 2025-06-27 07:42:45 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010125 or higher (current=1751010052); sleeping 5s (595s remaining) 2025-06-27 07:43:01 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010093, "numBadAcks": 0, "mdtTimestamp": 1751010052, "mdtTimestampMillisPart": 0} 2025-06-27 07:43:01 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010125 or higher (current=1751010052); sleeping 5s (590s remaining) 2025-06-27 07:43:18 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "818ae1449aded5cbf112ec9aa03f4a109140fb325be523e0252a9668bf59d451", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010186, "numBadAcks": 0, "mdtTimestamp": 1751010125, "mdtTimestampMillisPart": 0} 2025-06-27 07:43:18 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010125 2025-06-27 07:43:18 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010125 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:43:18 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.postfix1.channel1 with 15 files of size 1048576 bytes. 2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:43:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file 2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file 2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file 2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file 2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file 2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file 2025-06-27 07:43:19 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file 2025-06-27 07:43:19 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.postfix1.channel1 with 15 files of size 1048576 bytes. 2025-06-27 07:43:21 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix1.channel1?zone=1&submitType=full 2025-06-27 07:43:29 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix1.channel1 2025-06-27 07:43:30 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.postfix1.channel1.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 15, exceeding the configured limit of 10","requestID":"REQID-f9a2b7c0-001a-4825-9880-c36a85eea96b-mdt-test-8"}') 2025-06-27 07:43:30 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.postfix2.channel2 with 25 files of size 1048576 bytes. 2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:43:30 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_15.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_16.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_17.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_18.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_19.txt as a 1048576 byte random text file 2025-06-27 07:43:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_20.txt as a 1048576 byte random text file 2025-06-27 07:43:32 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_21.txt as a 1048576 byte random text file 2025-06-27 07:43:32 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_22.txt as a 1048576 byte random text file 2025-06-27 07:43:32 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_23.txt as a 1048576 byte random text file 2025-06-27 07:43:32 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_24.txt as a 1048576 byte random text file 2025-06-27 07:43:32 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.postfix2.channel2 with 25 files of size 1048576 bytes. 2025-06-27 07:43:33 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix2.channel2?zone=1&submitType=full 2025-06-27 07:43:43 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix2.channel2 2025-06-27 07:43:44 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.postfix2.channel2.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 25, exceeding the configured limit of 20","requestID":"REQID-c6355a08-f538-4eba-be50-c68a1842f471-mdt-test-8"}') 2025-06-27 07:43:44 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.postfix1.channel1 with 8 files of size 1048576 bytes. 2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:43:44 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:43:44 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.postfix1.channel1 with 8 files of size 1048576 bytes. 2025-06-27 07:43:46 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix1.channel1?zone=1&submitType=full 2025-06-27 07:43:53 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix1.channel1 2025-06-27 07:43:53 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.postfix1.channel1 with in channel limits is succeeded. 2025-06-27 07:43:53 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.postfix2.channel2 with 15 files of size 1048576 bytes. 2025-06-27 07:43:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:43:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:43:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:43:53 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file 2025-06-27 07:43:54 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.postfix2.channel2 with 15 files of size 1048576 bytes. 2025-06-27 07:43:56 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix2.channel2?zone=1&submitType=full 2025-06-27 07:44:04 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.postfix2.channel2 2025-06-27 07:44:04 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.postfix2.channel2 with in channel limits is succeeded. 2025-06-27 07:44:04 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_can_be_set_at_subnamespace' passed PASSED [ 83%] ------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------ 2025-06-27 07:44:04 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_can_be_set_at_subnamespace -> METANET-2542 2025-06-27 07:44:04 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation 2025-06-27 07:44:04 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:44:06 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:44:12 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:44:14 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010249.mdti2q-g 2025-06-27 07:44:18 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:44:20 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010249.mdti2q-g 2025-06-27 07:44:25 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:44:25 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010249 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:44:25 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010249 2025-06-27 07:44:36 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "818ae1449aded5cbf112ec9aa03f4a109140fb325be523e0252a9668bf59d451", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010186, "numBadAcks": 0, "mdtTimestamp": 1751010125, "mdtTimestampMillisPart": 0} 2025-06-27 07:44:36 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010249 or higher (current=1751010125); sleeping 5s (600s remaining) 2025-06-27 07:44:51 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010281, "numBadAcks": 0, "mdtTimestamp": 1751010249, "mdtTimestampMillisPart": 0} 2025-06-27 07:44:51 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010249 2025-06-27 07:44:51 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010249 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server tests/mdtreceiver/test_channel_limits_new.py::test_limits_can_be_set_per_channel ------------------------------------------------------------------------------------------------- live log setup -------------------------------------------------------------------------------------------------- 2025-06-27 07:44:51 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server -------------------------------------------------------------------------------------------------- live log call -------------------------------------------------------------------------------------------------- 2025-06-27 07:44:55 INFO root.pytest_runtest_call:28 - ==================== Start of test: 'test_limits_can_be_set_per_channel' ==================== 2025-06-27 07:44:55 DEBUG python_mdt.service.mdtreceiver.get_group_config:77 - Fetching current mdtreceiver group configuration from service=mdtreceiver, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:45:00 INFO test_channel_limits_new._test_runner:131 - Setting channel limits for multiple channels/namespaces. 2025-06-27 07:45:00 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*' 2025-06-27 07:45:00 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system. 2025-06-27 07:45:01 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.* 2025-06-27 07:45:01 INFO test_channel_limits_new._test_runner:135 - Test channel is test_mdt-test3.multi_2.dir.NbM8. 2025-06-27 07:45:01 INFO test_channel_limits_new._helper_to_update_granular_config:113 - Setting limits {'max_channel_file_count': 10, 'max_channel_mb': 20} for test_mdt-test3.multi_2.dir.NbM8. 2025-06-27 07:45:01 INFO root.test_channel:378 - Using MDT system name mdt-test3 to find channel by name '.multi_2.dir.*' 2025-06-27 07:45:01 INFO python_mdt.channel.mdtchannel.discover_channel_ns:298 - Search 'test_mdt-test3.multi_2.dir.*' in 'mdt-test3' MDT system. 2025-06-27 07:45:02 INFO python_mdt.channel.mdtchannel.discover_channel_ns:307 - Discovered MDT system namespace matching the channel partial name test_mdt-test3.multi_2.dir.* 2025-06-27 07:45:02 INFO test_channel_limits_new._test_runner:135 - Test channel is test_mdt-test3.multi_2.dir.1RXG. 2025-06-27 07:45:02 INFO test_channel_limits_new._helper_to_update_granular_config:113 - Setting limits {'max_channel_file_count': 20, 'max_channel_mb': 30} for test_mdt-test3.multi_2.dir.1RXG. 2025-06-27 07:45:02 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:45:04 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:45:09 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:45:11 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010307.mdti2q-g 2025-06-27 07:45:16 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:45:18 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010307.mdti2q-g 2025-06-27 07:45:23 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:45:23 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010307 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:45:23 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010307 2025-06-27 07:45:34 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010281, "numBadAcks": 0, "mdtTimestamp": 1751010249, "mdtTimestampMillisPart": 0} 2025-06-27 07:45:34 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010307 or higher (current=1751010249); sleeping 5s (600s remaining) 2025-06-27 07:45:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010281, "numBadAcks": 0, "mdtTimestamp": 1751010249, "mdtTimestampMillisPart": 0} 2025-06-27 07:45:49 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010307 or higher (current=1751010249); sleeping 5s (595s remaining) 2025-06-27 07:46:05 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "5b0c2c8b547864d6532256036a549f3989f7be8fbcda026ba1d159b53d92d8cf", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010363, "numBadAcks": 0, "mdtTimestamp": 1751010307, "mdtTimestampMillisPart": 0} 2025-06-27 07:46:05 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010307 2025-06-27 07:46:05 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010307 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:46:05 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.NbM8 with 15 files of size 1048576 bytes. 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file 2025-06-27 07:46:05 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file 2025-06-27 07:46:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file 2025-06-27 07:46:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file 2025-06-27 07:46:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file 2025-06-27 07:46:06 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file 2025-06-27 07:46:06 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.NbM8 with 15 files of size 1048576 bytes. 2025-06-27 07:46:07 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.NbM8?zone=1&submitType=full 2025-06-27 07:46:16 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.NbM8 2025-06-27 07:46:17 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.NbM8.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 15, exceeding the configured limit of 10","requestID":"REQID-c7c937f4-22b4-4bdd-bfb0-4bbe1d037be5-mdt-test-8"}') 2025-06-27 07:46:17 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.1RXG with 25 files of size 1048576 bytes. 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file 2025-06-27 07:46:17 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_15.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_16.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_17.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_18.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_19.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_20.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_21.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_22.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_23.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_24.txt as a 1048576 byte random text file 2025-06-27 07:46:18 INFO test_channel_limits_new._submission_helper_for_failures:66 - Submitting channel test_mdt-test3.multi_2.dir.1RXG with 25 files of size 1048576 bytes. 2025-06-27 07:46:20 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.1RXG?zone=1&submitType=full 2025-06-27 07:46:30 DEBUG python_alsi.machine.request:875 - Received 413 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.1RXG 2025-06-27 07:46:31 INFO test_channel_limits_new._submission_helper_for_failures:72 - Full submission for channel test_mdt-test3.multi_2.dir.1RXG.1 failed (status=413, message=b'{"type":"about:blank","title":"Exceeded max number of files in a channel","status":413,"detail":"Operation will result in a file count of 25, exceeding the configured limit of 20","requestID":"REQID-a0ae7dcf-c891-4827-a41e-b0a4e8e6643e-mdt-test-8"}') 2025-06-27 07:46:31 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.NbM8 with 8 files of size 1048576 bytes. 2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:46:31 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:46:31 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.NbM8 with 8 files of size 1048576 bytes. 2025-06-27 07:46:33 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.NbM8?zone=1&submitType=full 2025-06-27 07:46:40 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.NbM8 2025-06-27 07:46:40 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.NbM8 with in channel limits is succeeded. 2025-06-27 07:46:40 INFO test_channel_limits_new._prepare_content:36 - Preparing channel test_mdt-test3.multi_2.dir.1RXG with 15 files of size 1048576 bytes. 2025-06-27 07:46:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_0.txt as a 1048576 byte random text file 2025-06-27 07:46:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_1.txt as a 1048576 byte random text file 2025-06-27 07:46:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_2.txt as a 1048576 byte random text file 2025-06-27 07:46:40 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_3.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_4.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_5.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_6.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_7.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_8.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_9.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_10.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_11.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_12.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_13.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO python_mdt.channel.content.add_file_with_random_content:68 - Added file file_14.txt as a 1048576 byte random text file 2025-06-27 07:46:41 INFO test_channel_limits_new._submission_helper_for_success:88 - Submitting channel test_mdt-test3.multi_2.dir.1RXG with 15 files of size 1048576 bytes. 2025-06-27 07:46:43 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.1RXG?zone=1&submitType=full 2025-06-27 07:46:52 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/test_mdt-test3.multi_2.dir.1RXG 2025-06-27 07:46:53 INFO test_channel_limits_new._submission_helper_for_success:92 - Submission on channel test_mdt-test3.multi_2.dir.1RXG with in channel limits is succeeded. 2025-06-27 07:46:53 INFO root.pytest_runtest_makereport:52 - Test 'test_limits_can_be_set_per_channel' passed PASSED [100%] ------------------------------------------------------------------------------------------------ live log teardown ------------------------------------------------------------------------------------------------ 2025-06-27 07:46:53 DEBUG root.pytest_runtest_teardown:184 - Added Jira test ID test_limits_can_be_set_per_channel -> METANET-2539 2025-06-27 07:46:53 DEBUG root._cleanup_stack_generator:384 - Executing 1 function cleanup stack operation 2025-06-27 07:46:53 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:46:54 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:46:59 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:47:01 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010418.mdti2q-g 2025-06-27 07:47:06 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:47:08 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010418.mdti2q-g 2025-06-27 07:47:13 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:47:14 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010418 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:47:14 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010418 2025-06-27 07:47:24 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "5b0c2c8b547864d6532256036a549f3989f7be8fbcda026ba1d159b53d92d8cf", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010363, "numBadAcks": 0, "mdtTimestamp": 1751010307, "mdtTimestampMillisPart": 0} 2025-06-27 07:47:24 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010418 or higher (current=1751010307); sleeping 5s (600s remaining) 2025-06-27 07:47:40 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0} 2025-06-27 07:47:40 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010418 2025-06-27 07:47:40 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010418 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:47:45 INFO src.mdt.utils.backup_restore_files:139 - Restoring 'mdt.config.groups.mdt-test3'. { "granular_config": { "global_config": { "submit_to_zk": true } } } 2025-06-27 07:47:45 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.groups.mdt-test3' 2025-06-27 07:47:46 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=1&submitType=full 2025-06-27 07:47:51 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:47:53 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=2&uuid=mdt.config.groups.mdt-test3.1.0000001751010469.mdti2q-g 2025-06-27 07:47:57 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:47:59 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.groups.mdt-test3?zone=3&uuid=mdt.config.groups.mdt-test3.1.0000001751010469.mdti2q-g 2025-06-27 07:48:04 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.groups.mdt-test3 2025-06-27 07:48:04 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.groups.mdt-test3 has been updated to timestamp 1751010469 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:48:04 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.groups' for channel 'mdt.config.groups.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010469 2025-06-27 07:48:15 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0} 2025-06-27 07:48:15 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010469 or higher (current=1751010418); sleeping 5s (600s remaining) 2025-06-27 07:48:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0} 2025-06-27 07:48:30 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010469 or higher (current=1751010418); sleeping 5s (595s remaining) 2025-06-27 07:48:46 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0} 2025-06-27 07:48:46 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010469 or higher (current=1751010418); sleeping 5s (590s remaining) 2025-06-27 07:49:01 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010456, "numBadAcks": 0, "mdtTimestamp": 1751010418, "mdtTimestampMillisPart": 0} 2025-06-27 07:49:01 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.groups.mdt-test3 has not yet been updated to the requested timestamp 1751010469 or higher (current=1751010418); sleeping 5s (585s remaining) 2025-06-27 07:49:18 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.groups.mdt-test3' is: {"sha256": "be8cbfeb73711a42f99e2f4ecffce25cfe927811f4ed93fdb24e6c7ccdeb79d8", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.groups", "contentVersion": "1.3", "channel": "mdt.config.groups.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010550, "numBadAcks": 0, "mdtTimestamp": 1751010469, "mdtTimestampMillisPart": 0} 2025-06-27 07:49:18 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.groups.mdt-test3 is up to date with requested timestamp 1751010469 2025-06-27 07:49:18 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.groups.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010469 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:49:23 INFO src.mdt.utils.backup_restore_files:139 - Restoring 'mdt.config.mdtreceiver.mdt-test3'. { "enabled_jetstream_systems": [ "freeflow" ], "locking_installed_everywhere": false, "write_locking_mode": "ZK" } 2025-06-27 07:49:23 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:64 - Pushing JSON configuration to channel 'mdt.config.mdtreceiver.mdt-test3' 2025-06-27 07:49:25 DEBUG python_alsi.machine.http_request:845 - Issuing POST HTTP request with url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3?zone=1&submitType=full 2025-06-27 07:49:29 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3 2025-06-27 07:49:31 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3?zone=2&uuid=mdt.config.mdtreceiver.mdt-test3.1.0000001751010568.mdti2q-g 2025-06-27 07:49:36 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3 2025-06-27 07:49:38 DEBUG python_alsi.machine.http_request:845 - Issuing PUT HTTP request with url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3?zone=3&uuid=mdt.config.mdtreceiver.mdt-test3.1.0000001751010568.mdti2q-g 2025-06-27 07:49:42 DEBUG python_alsi.machine.request:875 - Received 201 HTTP response for request url=/mdt/v1/channels/mdt.config.mdtreceiver.mdt-test3 2025-06-27 07:49:43 INFO python_mdt.machine.mdtmachine.submit_and_verify_file_channel:67 - Verifying channel mdt.config.mdtreceiver.mdt-test3 has been updated to timestamp 1751010568 on MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:49:43 DEBUG python_mdt.service.mdtagent.get_persistent_state:279 - On machine 198.18.93.176, watching persistent state file '/a/mdtagent/var/persist/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver' for channel 'mdt.config.mdtreceiver.mdt-test3' to see that the channel has been updated to at least the latest timestamp 1751010568 2025-06-27 07:49:54 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.mdtreceiver.mdt-test3' is: {"sha256": "3689d7b7904acde8f3f9778f2ef2040d8966f8508910c42e2ce8b5a983b6cb78", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver", "contentVersion": "1.3", "channel": "mdt.config.mdtreceiver.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1750932299, "numBadAcks": 0, "mdtTimestamp": 1750932276, "mdtTimestampMillisPart": 0} 2025-06-27 07:49:54 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.mdtreceiver.mdt-test3 has not yet been updated to the requested timestamp 1751010568 or higher (current=1750932276); sleeping 5s (600s remaining) 2025-06-27 07:50:09 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.mdtreceiver.mdt-test3' is: {"sha256": "3689d7b7904acde8f3f9778f2ef2040d8966f8508910c42e2ce8b5a983b6cb78", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver", "contentVersion": "1.3", "channel": "mdt.config.mdtreceiver.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1750932299, "numBadAcks": 0, "mdtTimestamp": 1750932276, "mdtTimestampMillisPart": 0} 2025-06-27 07:50:09 DEBUG python_mdt.service.mdtagent.get_persistent_state:306 - Channel mdt.config.mdtreceiver.mdt-test3 has not yet been updated to the requested timestamp 1751010568 or higher (current=1750932276); sleeping 5s (595s remaining) 2025-06-27 07:50:25 DEBUG python_mdt.service.mdtagent.get_persistent_state:294 - Persistent state content for channel 'mdt.config.mdtreceiver.mdt-test3' is: {"sha256": "3689d7b7904acde8f3f9778f2ef2040d8966f8508910c42e2ce8b5a983b6cb78", "destFile": "/usr/local/akamai/mdtreceiver/etc/mdt.config.mdtreceiver", "contentVersion": "1.3", "channel": "mdt.config.mdtreceiver.mdt-test3", "group": "mdt_config", "zone": 2, "localTimestamp": 1751010612, "numBadAcks": 0, "mdtTimestamp": 1751010568, "mdtTimestampMillisPart": 0} 2025-06-27 07:50:25 INFO python_mdt.service.mdtagent.get_persistent_state:303 - Channel mdt.config.mdtreceiver.mdt-test3 is up to date with requested timestamp 1751010568 2025-06-27 07:50:25 INFO python_mdt.service.mdtagent.verify_channel_state:337 - Channel mdt.config.mdtreceiver.mdt-test3 was downloaded (and optionally notified) successfully for timestamp 1751010568 on service=mdtagent, MdtServerMachine: ip=198.18.93.176, network=metanet, testnet=sqa2, role=server 2025-06-27 07:50:25 DEBUG root.delete_paths:344 - Deleting temporary paths. ========================================================================================= 6 passed in 1721.82s (0:28:41) ==========================================================================================