Adding Conv-TasNet

[mthrok]

This issue is for tracking the PRs for Conv-TasNet training pipeline.

PRs for prototype

• Add conv-tasnet model #893 -> Add ConvTasNet model #920 Model
• Add si_snr metrics #894 Metrics
• Add wsj0mix dataset #895 wsj0mix dataset
• Add conv-tasnet training script #896 training script

Follow up

• Update citation Add si_snr metrics #894 (comment)
• Add pretrained model
  • Retrain the model
  • Upload pre-trained model
  • Add code for pre-trained model
• Move C++ runtime to master branch: See my branch
• Move metrics/loss to library.

Others

• Change loss to SI-SNR and re-train.
• notebook-tize
• Add google colab

[mthrok]

FYI: Initial training result

Best evaluation SI_SNR: 11.22 (Epoch: 54)

Some separation result (from top to bottom, mixed signal, source 1, source 2, separated speech 1, separated speech 2)

log

2020-08-27 08:53:24,322: [worker  0] ----------------------------------------------------------------------
2020-08-27 08:53:24,322: [worker  0] Epoch: 1
2020-08-27 08:53:24,322: [worker  0] Learning rate: 0.001
2020-08-27 08:53:24,322: [worker  0] ----------------------------------------------------------------------
2020-08-27 08:57:20,850: [worker  0] Eval   Loss: -1.437e+00
2020-08-27 08:57:20,850: [worker  0] Eval  Speed: 165.30 [samples/sec]
2020-08-27 08:57:21,148: [worker  0] ----------------------------------------------------------------------
2020-08-27 08:57:21,148: [worker  0] Epoch: 2
2020-08-27 08:57:21,148: [worker  0] Learning rate: 0.001
2020-08-27 08:57:21,148: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:00:31,881: [worker  0] Eval   Loss: -2.280e+00
2020-08-27 09:00:31,881: [worker  0] Eval  Speed: 254.57 [samples/sec]
2020-08-27 09:00:32,121: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:00:32,121: [worker  0] Epoch: 3
2020-08-27 09:00:32,121: [worker  0] Learning rate: 0.001
2020-08-27 09:00:32,121: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:03:43,234: [worker  0] Eval   Loss: -4.470e+00
2020-08-27 09:03:43,234: [worker  0] Eval  Speed: 252.24 [samples/sec]
2020-08-27 09:03:43,579: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:03:43,580: [worker  0] Epoch: 4
2020-08-27 09:03:43,580: [worker  0] Learning rate: 0.001
2020-08-27 09:03:43,580: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:06:55,924: [worker  0] Eval   Loss: -5.678e+00
2020-08-27 09:06:55,925: [worker  0] Eval  Speed: 253.08 [samples/sec]
2020-08-27 09:06:56,216: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:06:56,216: [worker  0] Epoch: 5
2020-08-27 09:06:56,216: [worker  0] Learning rate: 0.001
2020-08-27 09:06:56,216: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:10:08,578: [worker  0] Eval   Loss: -5.935e+00
2020-08-27 09:10:08,578: [worker  0] Eval  Speed: 252.42 [samples/sec]
2020-08-27 09:10:08,926: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:10:08,927: [worker  0] Epoch: 6
2020-08-27 09:10:08,927: [worker  0] Learning rate: 0.001
2020-08-27 09:10:08,927: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:13:21,603: [worker  0] Eval   Loss: -6.443e+00
2020-08-27 09:13:21,603: [worker  0] Eval  Speed: 254.78 [samples/sec]
2020-08-27 09:13:21,880: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:13:21,880: [worker  0] Epoch: 7
2020-08-27 09:13:21,880: [worker  0] Learning rate: 0.001
2020-08-27 09:13:21,880: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:16:35,345: [worker  0] Eval   Loss: -6.236e+00
2020-08-27 09:16:35,346: [worker  0] Eval  Speed: 255.17 [samples/sec]
2020-08-27 09:16:35,646: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:16:35,646: [worker  0] Epoch: 8
2020-08-27 09:16:35,646: [worker  0] Learning rate: 0.001
2020-08-27 09:16:35,646: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:19:48,200: [worker  0] Eval   Loss: -7.004e+00
2020-08-27 09:19:48,200: [worker  0] Eval  Speed: 251.78 [samples/sec]
2020-08-27 09:19:48,453: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:19:48,453: [worker  0] Epoch: 9
2020-08-27 09:19:48,453: [worker  0] Learning rate: 0.001
2020-08-27 09:19:48,453: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:23:02,045: [worker  0] Eval   Loss: -7.599e+00
2020-08-27 09:23:02,045: [worker  0] Eval  Speed: 250.70 [samples/sec]
2020-08-27 09:23:02,296: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:23:02,296: [worker  0] Epoch: 10
2020-08-27 09:23:02,296: [worker  0] Learning rate: 0.001
2020-08-27 09:23:02,296: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:26:15,763: [worker  0] Eval   Loss: -7.830e+00
2020-08-27 09:26:15,763: [worker  0] Eval  Speed: 252.50 [samples/sec]
2020-08-27 09:26:15,996: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:26:15,997: [worker  0] Epoch: 11
2020-08-27 09:26:15,997: [worker  0] Learning rate: 0.001
2020-08-27 09:26:15,997: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:29:29,614: [worker  0] Eval   Loss: -8.178e+00
2020-08-27 09:29:29,615: [worker  0] Eval  Speed: 253.23 [samples/sec]
2020-08-27 09:29:29,940: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:29:29,940: [worker  0] Epoch: 12
2020-08-27 09:29:29,940: [worker  0] Learning rate: 0.001
2020-08-27 09:29:29,940: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:32:43,168: [worker  0] Eval   Loss: -8.156e+00
2020-08-27 09:32:43,168: [worker  0] Eval  Speed: 252.71 [samples/sec]
2020-08-27 09:32:43,435: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:32:43,435: [worker  0] Epoch: 13
2020-08-27 09:32:43,435: [worker  0] Learning rate: 0.001
2020-08-27 09:32:43,435: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:35:57,546: [worker  0] Eval   Loss: -8.621e+00
2020-08-27 09:35:57,546: [worker  0] Eval  Speed: 243.12 [samples/sec]
2020-08-27 09:35:57,840: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:35:57,840: [worker  0] Epoch: 14
2020-08-27 09:35:57,840: [worker  0] Learning rate: 0.001
2020-08-27 09:35:57,840: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:39:11,999: [worker  0] Eval   Loss: -8.858e+00
2020-08-27 09:39:11,999: [worker  0] Eval  Speed: 253.50 [samples/sec]
2020-08-27 09:39:12,322: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:39:12,322: [worker  0] Epoch: 15
2020-08-27 09:39:12,322: [worker  0] Learning rate: 0.001
2020-08-27 09:39:12,322: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:42:25,624: [worker  0] Eval   Loss: -8.279e+00
2020-08-27 09:42:25,624: [worker  0] Eval  Speed: 251.64 [samples/sec]
2020-08-27 09:42:25,896: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:42:25,896: [worker  0] Epoch: 16
2020-08-27 09:42:25,896: [worker  0] Learning rate: 0.001
2020-08-27 09:42:25,896: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:45:40,415: [worker  0] Eval   Loss: -8.468e+00
2020-08-27 09:45:40,415: [worker  0] Eval  Speed: 234.62 [samples/sec]
2020-08-27 09:45:40,746: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:45:40,746: [worker  0] Epoch: 17
2020-08-27 09:45:40,746: [worker  0] Learning rate: 0.001
2020-08-27 09:45:40,746: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:48:55,263: [worker  0] Eval   Loss: -8.575e+00
2020-08-27 09:48:55,263: [worker  0] Eval  Speed: 244.19 [samples/sec]
2020-08-27 09:48:55,688: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:48:55,688: [worker  0] Epoch: 18
2020-08-27 09:48:55,688: [worker  0] Learning rate: 0.001
2020-08-27 09:48:55,688: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:52:09,786: [worker  0] Eval   Loss: -9.340e+00
2020-08-27 09:52:09,786: [worker  0] Eval  Speed: 249.71 [samples/sec]
2020-08-27 09:52:10,162: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:52:10,163: [worker  0] Epoch: 19
2020-08-27 09:52:10,163: [worker  0] Learning rate: 0.001
2020-08-27 09:52:10,163: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:55:23,193: [worker  0] Eval   Loss: -8.736e+00
2020-08-27 09:55:23,193: [worker  0] Eval  Speed: 251.68 [samples/sec]
2020-08-27 09:55:23,434: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:55:23,434: [worker  0] Epoch: 20
2020-08-27 09:55:23,434: [worker  0] Learning rate: 0.001
2020-08-27 09:55:23,434: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:58:36,373: [worker  0] Eval   Loss: -9.161e+00
2020-08-27 09:58:36,373: [worker  0] Eval  Speed: 251.39 [samples/sec]
2020-08-27 09:58:36,716: [worker  0] ----------------------------------------------------------------------
2020-08-27 09:58:36,716: [worker  0] Epoch: 21
2020-08-27 09:58:36,716: [worker  0] Learning rate: 0.001
2020-08-27 09:58:36,716: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:01:50,343: [worker  0] Eval   Loss: -9.298e+00
2020-08-27 10:01:50,343: [worker  0] Eval  Speed: 253.71 [samples/sec]
2020-08-27 10:01:50,580: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:01:50,580: [worker  0] Epoch: 22
2020-08-27 10:01:50,580: [worker  0] Learning rate: 0.001
2020-08-27 10:01:50,581: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:05:03,327: [worker  0] Eval   Loss: -8.873e+00
2020-08-27 10:05:03,327: [worker  0] Eval  Speed: 250.91 [samples/sec]
2020-08-27 10:05:03,826: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:05:03,826: [worker  0] Epoch: 23
2020-08-27 10:05:03,826: [worker  0] Learning rate: 0.0005
2020-08-27 10:05:03,826: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:08:15,955: [worker  0] Eval   Loss: -1.003e+01
2020-08-27 10:08:15,955: [worker  0] Eval  Speed: 254.88 [samples/sec]
2020-08-27 10:08:16,257: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:08:16,258: [worker  0] Epoch: 24
2020-08-27 10:08:16,258: [worker  0] Learning rate: 0.0005
2020-08-27 10:08:16,258: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:11:27,461: [worker  0] Eval   Loss: -9.854e+00
2020-08-27 10:11:27,462: [worker  0] Eval  Speed: 249.47 [samples/sec]
2020-08-27 10:11:27,893: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:11:27,893: [worker  0] Epoch: 25
2020-08-27 10:11:27,893: [worker  0] Learning rate: 0.0005
2020-08-27 10:11:27,893: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:14:39,589: [worker  0] Eval   Loss: -9.866e+00
2020-08-27 10:14:39,589: [worker  0] Eval  Speed: 252.70 [samples/sec]
2020-08-27 10:14:39,893: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:14:39,893: [worker  0] Epoch: 26
2020-08-27 10:14:39,893: [worker  0] Learning rate: 0.0005
2020-08-27 10:14:39,893: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:17:51,384: [worker  0] Eval   Loss: -1.009e+01
2020-08-27 10:17:51,384: [worker  0] Eval  Speed: 252.38 [samples/sec]
2020-08-27 10:17:51,639: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:17:51,639: [worker  0] Epoch: 27
2020-08-27 10:17:51,639: [worker  0] Learning rate: 0.0005
2020-08-27 10:17:51,639: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:21:02,276: [worker  0] Eval   Loss: -1.003e+01
2020-08-27 10:21:02,276: [worker  0] Eval  Speed: 251.92 [samples/sec]
2020-08-27 10:21:02,578: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:21:02,578: [worker  0] Epoch: 28
2020-08-27 10:21:02,578: [worker  0] Learning rate: 0.0005
2020-08-27 10:21:02,578: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:24:13,469: [worker  0] Eval   Loss: -1.016e+01
2020-08-27 10:24:13,469: [worker  0] Eval  Speed: 250.15 [samples/sec]
2020-08-27 10:24:13,785: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:24:13,785: [worker  0] Epoch: 29
2020-08-27 10:24:13,785: [worker  0] Learning rate: 0.0005
2020-08-27 10:24:13,785: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:27:23,838: [worker  0] Eval   Loss: -1.022e+01
2020-08-27 10:27:23,838: [worker  0] Eval  Speed: 252.97 [samples/sec]
2020-08-27 10:27:24,126: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:27:24,126: [worker  0] Epoch: 30
2020-08-27 10:27:24,126: [worker  0] Learning rate: 0.0005
2020-08-27 10:27:24,127: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:30:34,807: [worker  0] Eval   Loss: -1.021e+01
2020-08-27 10:30:34,807: [worker  0] Eval  Speed: 252.64 [samples/sec]
2020-08-27 10:30:35,083: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:30:35,083: [worker  0] Epoch: 31
2020-08-27 10:30:35,083: [worker  0] Learning rate: 0.0005
2020-08-27 10:30:35,083: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:33:46,272: [worker  0] Eval   Loss: -1.042e+01
2020-08-27 10:33:46,272: [worker  0] Eval  Speed: 255.60 [samples/sec]
2020-08-27 10:33:46,641: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:33:46,641: [worker  0] Epoch: 32
2020-08-27 10:33:46,641: [worker  0] Learning rate: 0.0005
2020-08-27 10:33:46,641: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:36:57,126: [worker  0] Eval   Loss: -1.049e+01
2020-08-27 10:36:57,126: [worker  0] Eval  Speed: 252.65 [samples/sec]
2020-08-27 10:36:57,497: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:36:57,497: [worker  0] Epoch: 33
2020-08-27 10:36:57,497: [worker  0] Learning rate: 0.0005
2020-08-27 10:36:57,497: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:40:08,672: [worker  0] Eval   Loss: -1.013e+01
2020-08-27 10:40:08,672: [worker  0] Eval  Speed: 233.33 [samples/sec]
2020-08-27 10:40:09,017: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:40:09,017: [worker  0] Epoch: 34
2020-08-27 10:40:09,017: [worker  0] Learning rate: 0.0005
2020-08-27 10:40:09,017: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:43:19,629: [worker  0] Eval   Loss: -1.042e+01
2020-08-27 10:43:19,629: [worker  0] Eval  Speed: 252.83 [samples/sec]
2020-08-27 10:43:20,046: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:43:20,046: [worker  0] Epoch: 35
2020-08-27 10:43:20,046: [worker  0] Learning rate: 0.0005
2020-08-27 10:43:20,046: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:46:30,742: [worker  0] Eval   Loss: -1.008e+01
2020-08-27 10:46:30,742: [worker  0] Eval  Speed: 252.83 [samples/sec]
2020-08-27 10:46:31,085: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:46:31,086: [worker  0] Epoch: 36
2020-08-27 10:46:31,086: [worker  0] Learning rate: 0.0005
2020-08-27 10:46:31,086: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:49:41,071: [worker  0] Eval   Loss: -1.023e+01
2020-08-27 10:49:41,071: [worker  0] Eval  Speed: 252.54 [samples/sec]
2020-08-27 10:49:41,355: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:49:41,355: [worker  0] Epoch: 37
2020-08-27 10:49:41,355: [worker  0] Learning rate: 0.00025
2020-08-27 10:49:41,355: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:52:51,211: [worker  0] Eval   Loss: -1.065e+01
2020-08-27 10:52:51,211: [worker  0] Eval  Speed: 254.14 [samples/sec]
2020-08-27 10:52:51,604: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:52:51,604: [worker  0] Epoch: 38
2020-08-27 10:52:51,604: [worker  0] Learning rate: 0.00025
2020-08-27 10:52:51,604: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:56:01,888: [worker  0] Eval   Loss: -1.100e+01
2020-08-27 10:56:01,888: [worker  0] Eval  Speed: 251.14 [samples/sec]
2020-08-27 10:56:02,266: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:56:02,266: [worker  0] Epoch: 39
2020-08-27 10:56:02,266: [worker  0] Learning rate: 0.00025
2020-08-27 10:56:02,267: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:59:12,656: [worker  0] Eval   Loss: -1.102e+01
2020-08-27 10:59:12,656: [worker  0] Eval  Speed: 251.54 [samples/sec]
2020-08-27 10:59:12,969: [worker  0] ----------------------------------------------------------------------
2020-08-27 10:59:12,969: [worker  0] Epoch: 40
2020-08-27 10:59:12,969: [worker  0] Learning rate: 0.00025
2020-08-27 10:59:12,969: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:02:22,547: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 11:02:22,547: [worker  0] Eval  Speed: 255.97 [samples/sec]
2020-08-27 11:02:22,961: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:02:22,961: [worker  0] Epoch: 41
2020-08-27 11:02:22,961: [worker  0] Learning rate: 0.00025
2020-08-27 11:02:22,961: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:05:32,799: [worker  0] Eval   Loss: -1.111e+01
2020-08-27 11:05:32,799: [worker  0] Eval  Speed: 249.90 [samples/sec]
2020-08-27 11:05:33,193: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:05:33,193: [worker  0] Epoch: 42
2020-08-27 11:05:33,193: [worker  0] Learning rate: 0.00025
2020-08-27 11:05:33,193: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:08:44,043: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 11:08:44,043: [worker  0] Eval  Speed: 256.59 [samples/sec]
2020-08-27 11:08:44,360: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:08:44,360: [worker  0] Epoch: 43
2020-08-27 11:08:44,360: [worker  0] Learning rate: 0.00025
2020-08-27 11:08:44,360: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:11:54,087: [worker  0] Eval   Loss: -1.111e+01
2020-08-27 11:11:54,087: [worker  0] Eval  Speed: 255.69 [samples/sec]
2020-08-27 11:11:54,386: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:11:54,387: [worker  0] Epoch: 44
2020-08-27 11:11:54,387: [worker  0] Learning rate: 0.00025
2020-08-27 11:11:54,387: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:15:04,231: [worker  0] Eval   Loss: -1.103e+01
2020-08-27 11:15:04,231: [worker  0] Eval  Speed: 254.48 [samples/sec]
2020-08-27 11:15:04,548: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:15:04,548: [worker  0] Epoch: 45
2020-08-27 11:15:04,548: [worker  0] Learning rate: 0.00025
2020-08-27 11:15:04,548: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:18:16,341: [worker  0] Eval   Loss: -1.116e+01
2020-08-27 11:18:16,341: [worker  0] Eval  Speed: 248.57 [samples/sec]
2020-08-27 11:18:16,736: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:18:16,737: [worker  0] Epoch: 46
2020-08-27 11:18:16,737: [worker  0] Learning rate: 0.00025
2020-08-27 11:18:16,737: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:21:28,206: [worker  0] Eval   Loss: -1.113e+01
2020-08-27 11:21:28,206: [worker  0] Eval  Speed: 251.47 [samples/sec]
2020-08-27 11:21:28,504: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:21:28,504: [worker  0] Epoch: 47
2020-08-27 11:21:28,504: [worker  0] Learning rate: 0.00025
2020-08-27 11:21:28,504: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:24:39,431: [worker  0] Eval   Loss: -1.100e+01
2020-08-27 11:24:39,432: [worker  0] Eval  Speed: 252.65 [samples/sec]
2020-08-27 11:24:39,758: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:24:39,758: [worker  0] Epoch: 48
2020-08-27 11:24:39,758: [worker  0] Learning rate: 0.00025
2020-08-27 11:24:39,758: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:27:50,081: [worker  0] Eval   Loss: -1.097e+01
2020-08-27 11:27:50,081: [worker  0] Eval  Speed: 252.72 [samples/sec]
2020-08-27 11:27:50,467: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:27:50,467: [worker  0] Epoch: 49
2020-08-27 11:27:50,467: [worker  0] Learning rate: 0.00025
2020-08-27 11:27:50,467: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:31:01,097: [worker  0] Eval   Loss: -1.112e+01
2020-08-27 11:31:01,097: [worker  0] Eval  Speed: 255.52 [samples/sec]
2020-08-27 11:31:01,386: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:31:01,386: [worker  0] Epoch: 50
2020-08-27 11:31:01,386: [worker  0] Learning rate: 0.000125
2020-08-27 11:31:01,386: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:34:12,343: [worker  0] Eval   Loss: -1.113e+01
2020-08-27 11:34:12,343: [worker  0] Eval  Speed: 246.50 [samples/sec]
2020-08-27 11:34:12,639: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:34:12,639: [worker  0] Epoch: 51
2020-08-27 11:34:12,639: [worker  0] Learning rate: 0.000125
2020-08-27 11:34:12,639: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:37:23,780: [worker  0] Eval   Loss: -1.110e+01
2020-08-27 11:37:23,781: [worker  0] Eval  Speed: 253.39 [samples/sec]
2020-08-27 11:37:24,076: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:37:24,076: [worker  0] Epoch: 52
2020-08-27 11:37:24,076: [worker  0] Learning rate: 0.000125
2020-08-27 11:37:24,077: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:40:34,683: [worker  0] Eval   Loss: -1.110e+01
2020-08-27 11:40:34,683: [worker  0] Eval  Speed: 251.88 [samples/sec]
2020-08-27 11:40:35,148: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:40:35,148: [worker  0] Epoch: 53
2020-08-27 11:40:35,149: [worker  0] Learning rate: 0.000125
2020-08-27 11:40:35,149: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:43:46,333: [worker  0] Eval   Loss: -1.110e+01
2020-08-27 11:43:46,333: [worker  0] Eval  Speed: 251.13 [samples/sec]
2020-08-27 11:43:46,611: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:43:46,611: [worker  0] Epoch: 54
2020-08-27 11:43:46,611: [worker  0] Learning rate: 6.25e-05
2020-08-27 11:43:46,611: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:46:57,170: [worker  0] Eval   Loss: -1.122e+01
2020-08-27 11:46:57,170: [worker  0] Eval  Speed: 251.96 [samples/sec]
2020-08-27 11:46:57,706: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:46:57,706: [worker  0] Epoch: 55
2020-08-27 11:46:57,706: [worker  0] Learning rate: 6.25e-05
2020-08-27 11:46:57,706: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:50:08,158: [worker  0] Eval   Loss: -1.121e+01
2020-08-27 11:50:08,158: [worker  0] Eval  Speed: 252.59 [samples/sec]
2020-08-27 11:50:08,490: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:50:08,490: [worker  0] Epoch: 56
2020-08-27 11:50:08,490: [worker  0] Learning rate: 6.25e-05
2020-08-27 11:50:08,490: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:53:22,531: [worker  0] Eval   Loss: -1.119e+01
2020-08-27 11:53:22,531: [worker  0] Eval  Speed: 234.12 [samples/sec]
2020-08-27 11:53:22,889: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:53:22,890: [worker  0] Epoch: 57
2020-08-27 11:53:22,890: [worker  0] Learning rate: 6.25e-05
2020-08-27 11:53:22,890: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:56:33,685: [worker  0] Eval   Loss: -1.118e+01
2020-08-27 11:56:33,686: [worker  0] Eval  Speed: 252.73 [samples/sec]
2020-08-27 11:56:34,004: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:56:34,004: [worker  0] Epoch: 58
2020-08-27 11:56:34,004: [worker  0] Learning rate: 6.25e-05
2020-08-27 11:56:34,004: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:59:44,856: [worker  0] Eval   Loss: -1.116e+01
2020-08-27 11:59:44,856: [worker  0] Eval  Speed: 240.72 [samples/sec]
2020-08-27 11:59:45,281: [worker  0] ----------------------------------------------------------------------
2020-08-27 11:59:45,281: [worker  0] Epoch: 59
2020-08-27 11:59:45,281: [worker  0] Learning rate: 3.125e-05
2020-08-27 11:59:45,281: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:03:04,900: [worker  0] Eval   Loss: -1.114e+01
2020-08-27 12:03:04,900: [worker  0] Eval  Speed: 252.50 [samples/sec]
2020-08-27 12:03:05,368: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:03:05,368: [worker  0] Epoch: 60
2020-08-27 12:03:05,368: [worker  0] Learning rate: 3.125e-05
2020-08-27 12:03:05,368: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:06:16,796: [worker  0] Eval   Loss: -1.111e+01
2020-08-27 12:06:16,797: [worker  0] Eval  Speed: 252.70 [samples/sec]
2020-08-27 12:06:17,202: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:06:17,202: [worker  0] Epoch: 61
2020-08-27 12:06:17,202: [worker  0] Learning rate: 3.125e-05
2020-08-27 12:06:17,202: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:09:27,658: [worker  0] Eval   Loss: -1.109e+01
2020-08-27 12:09:27,658: [worker  0] Eval  Speed: 254.51 [samples/sec]
2020-08-27 12:09:28,005: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:09:28,005: [worker  0] Epoch: 62
2020-08-27 12:09:28,005: [worker  0] Learning rate: 3.125e-05
2020-08-27 12:09:28,005: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:12:38,953: [worker  0] Eval   Loss: -1.108e+01
2020-08-27 12:12:38,954: [worker  0] Eval  Speed: 252.27 [samples/sec]
2020-08-27 12:12:39,421: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:12:39,421: [worker  0] Epoch: 63
2020-08-27 12:12:39,421: [worker  0] Learning rate: 1.5625e-05
2020-08-27 12:12:39,421: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:15:50,581: [worker  0] Eval   Loss: -1.099e+01
2020-08-27 12:15:50,581: [worker  0] Eval  Speed: 252.48 [samples/sec]
2020-08-27 12:15:50,904: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:15:50,904: [worker  0] Epoch: 64
2020-08-27 12:15:50,904: [worker  0] Learning rate: 1.5625e-05
2020-08-27 12:15:50,905: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:19:01,645: [worker  0] Eval   Loss: -1.100e+01
2020-08-27 12:19:01,645: [worker  0] Eval  Speed: 251.86 [samples/sec]
2020-08-27 12:19:02,010: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:19:02,010: [worker  0] Epoch: 65
2020-08-27 12:19:02,010: [worker  0] Learning rate: 1.5625e-05
2020-08-27 12:19:02,010: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:22:12,589: [worker  0] Eval   Loss: -1.100e+01
2020-08-27 12:22:12,589: [worker  0] Eval  Speed: 253.46 [samples/sec]
2020-08-27 12:22:12,941: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:22:12,941: [worker  0] Epoch: 66
2020-08-27 12:22:12,941: [worker  0] Learning rate: 1.5625e-05
2020-08-27 12:22:12,941: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:25:23,780: [worker  0] Eval   Loss: -1.100e+01
2020-08-27 12:25:23,780: [worker  0] Eval  Speed: 249.15 [samples/sec]
2020-08-27 12:25:24,102: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:25:24,102: [worker  0] Epoch: 67
2020-08-27 12:25:24,102: [worker  0] Learning rate: 7.8125e-06
2020-08-27 12:25:24,102: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:28:35,158: [worker  0] Eval   Loss: -1.103e+01
2020-08-27 12:28:35,159: [worker  0] Eval  Speed: 241.75 [samples/sec]
2020-08-27 12:28:35,576: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:28:35,576: [worker  0] Epoch: 68
2020-08-27 12:28:35,576: [worker  0] Learning rate: 7.8125e-06
2020-08-27 12:28:35,576: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:31:49,949: [worker  0] Eval   Loss: -1.103e+01
2020-08-27 12:31:49,950: [worker  0] Eval  Speed: 251.17 [samples/sec]
2020-08-27 12:31:50,282: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:31:50,282: [worker  0] Epoch: 69
2020-08-27 12:31:50,282: [worker  0] Learning rate: 7.8125e-06
2020-08-27 12:31:50,282: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:35:00,551: [worker  0] Eval   Loss: -1.103e+01
2020-08-27 12:35:00,551: [worker  0] Eval  Speed: 254.09 [samples/sec]
2020-08-27 12:35:00,872: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:35:00,872: [worker  0] Epoch: 70
2020-08-27 12:35:00,872: [worker  0] Learning rate: 7.8125e-06
2020-08-27 12:35:00,872: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:38:11,488: [worker  0] Eval   Loss: -1.103e+01
2020-08-27 12:38:11,489: [worker  0] Eval  Speed: 251.89 [samples/sec]
2020-08-27 12:38:11,831: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:38:11,831: [worker  0] Epoch: 71
2020-08-27 12:38:11,831: [worker  0] Learning rate: 3.90625e-06
2020-08-27 12:38:11,831: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:41:23,019: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 12:41:23,020: [worker  0] Eval  Speed: 253.90 [samples/sec]
2020-08-27 12:41:23,433: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:41:23,433: [worker  0] Epoch: 72
2020-08-27 12:41:23,434: [worker  0] Learning rate: 3.90625e-06
2020-08-27 12:41:23,434: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:44:33,969: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 12:44:33,969: [worker  0] Eval  Speed: 248.12 [samples/sec]
2020-08-27 12:44:34,276: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:44:34,276: [worker  0] Epoch: 73
2020-08-27 12:44:34,276: [worker  0] Learning rate: 3.90625e-06
2020-08-27 12:44:34,276: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:47:44,938: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 12:47:44,938: [worker  0] Eval  Speed: 254.75 [samples/sec]
2020-08-27 12:47:45,323: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:47:45,323: [worker  0] Epoch: 74
2020-08-27 12:47:45,323: [worker  0] Learning rate: 3.90625e-06
2020-08-27 12:47:45,324: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:50:56,029: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 12:50:56,029: [worker  0] Eval  Speed: 253.32 [samples/sec]
2020-08-27 12:50:56,338: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:50:56,338: [worker  0] Epoch: 75
2020-08-27 12:50:56,338: [worker  0] Learning rate: 1.953125e-06
2020-08-27 12:50:56,338: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:54:07,099: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 12:54:07,099: [worker  0] Eval  Speed: 249.98 [samples/sec]
2020-08-27 12:54:07,445: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:54:07,445: [worker  0] Epoch: 76
2020-08-27 12:54:07,445: [worker  0] Learning rate: 1.953125e-06
2020-08-27 12:54:07,445: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:57:17,739: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 12:57:17,739: [worker  0] Eval  Speed: 253.24 [samples/sec]
2020-08-27 12:57:18,047: [worker  0] ----------------------------------------------------------------------
2020-08-27 12:57:18,047: [worker  0] Epoch: 77
2020-08-27 12:57:18,047: [worker  0] Learning rate: 1.953125e-06
2020-08-27 12:57:18,047: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:00:28,750: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 13:00:28,750: [worker  0] Eval  Speed: 249.94 [samples/sec]
2020-08-27 13:00:29,062: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:00:29,062: [worker  0] Epoch: 78
2020-08-27 13:00:29,062: [worker  0] Learning rate: 1.953125e-06
2020-08-27 13:00:29,062: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:03:39,334: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 13:03:39,334: [worker  0] Eval  Speed: 253.14 [samples/sec]
2020-08-27 13:03:39,614: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:03:39,614: [worker  0] Epoch: 79
2020-08-27 13:03:39,614: [worker  0] Learning rate: 9.765625e-07
2020-08-27 13:03:39,614: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:06:49,909: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 13:06:49,909: [worker  0] Eval  Speed: 253.46 [samples/sec]
2020-08-27 13:06:50,495: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:06:50,495: [worker  0] Epoch: 80
2020-08-27 13:06:50,495: [worker  0] Learning rate: 9.765625e-07
2020-08-27 13:06:50,495: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:10:00,738: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 13:10:00,738: [worker  0] Eval  Speed: 254.12 [samples/sec]
2020-08-27 13:10:01,014: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:10:01,014: [worker  0] Epoch: 81
2020-08-27 13:10:01,014: [worker  0] Learning rate: 9.765625e-07
2020-08-27 13:10:01,014: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:13:12,391: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 13:13:12,391: [worker  0] Eval  Speed: 254.01 [samples/sec]
2020-08-27 13:13:12,788: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:13:12,788: [worker  0] Epoch: 82
2020-08-27 13:13:12,788: [worker  0] Learning rate: 9.765625e-07
2020-08-27 13:13:12,788: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:16:23,561: [worker  0] Eval   Loss: -1.106e+01
2020-08-27 13:16:23,561: [worker  0] Eval  Speed: 252.30 [samples/sec]
2020-08-27 13:16:23,933: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:16:23,933: [worker  0] Epoch: 83
2020-08-27 13:16:23,933: [worker  0] Learning rate: 4.8828125e-07
2020-08-27 13:16:23,933: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:19:34,416: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:19:34,417: [worker  0] Eval  Speed: 252.31 [samples/sec]
2020-08-27 13:19:34,760: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:19:34,760: [worker  0] Epoch: 84
2020-08-27 13:19:34,760: [worker  0] Learning rate: 4.8828125e-07
2020-08-27 13:19:34,760: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:22:44,872: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:22:44,872: [worker  0] Eval  Speed: 253.86 [samples/sec]
2020-08-27 13:22:45,161: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:22:45,162: [worker  0] Epoch: 85
2020-08-27 13:22:45,162: [worker  0] Learning rate: 4.8828125e-07
2020-08-27 13:22:45,162: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:25:55,315: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:25:55,316: [worker  0] Eval  Speed: 254.97 [samples/sec]
2020-08-27 13:25:55,669: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:25:55,669: [worker  0] Epoch: 86
2020-08-27 13:25:55,669: [worker  0] Learning rate: 4.8828125e-07
2020-08-27 13:25:55,670: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:29:06,126: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:29:06,127: [worker  0] Eval  Speed: 250.50 [samples/sec]
2020-08-27 13:29:06,400: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:29:06,400: [worker  0] Epoch: 87
2020-08-27 13:29:06,400: [worker  0] Learning rate: 2.44140625e-07
2020-08-27 13:29:06,400: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:32:17,044: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:32:17,044: [worker  0] Eval  Speed: 253.35 [samples/sec]
2020-08-27 13:32:17,329: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:32:17,330: [worker  0] Epoch: 88
2020-08-27 13:32:17,330: [worker  0] Learning rate: 2.44140625e-07
2020-08-27 13:32:17,330: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:35:27,665: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:35:27,666: [worker  0] Eval  Speed: 252.39 [samples/sec]
2020-08-27 13:35:28,054: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:35:28,054: [worker  0] Epoch: 89
2020-08-27 13:35:28,054: [worker  0] Learning rate: 2.44140625e-07
2020-08-27 13:35:28,054: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:38:38,773: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:38:38,774: [worker  0] Eval  Speed: 250.70 [samples/sec]
2020-08-27 13:38:39,065: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:38:39,065: [worker  0] Epoch: 90
2020-08-27 13:38:39,065: [worker  0] Learning rate: 2.44140625e-07
2020-08-27 13:38:39,065: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:41:49,687: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:41:49,687: [worker  0] Eval  Speed: 253.32 [samples/sec]
2020-08-27 13:41:49,961: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:41:49,961: [worker  0] Epoch: 91
2020-08-27 13:41:49,961: [worker  0] Learning rate: 1.220703125e-07
2020-08-27 13:41:49,961: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:45:00,453: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:45:00,453: [worker  0] Eval  Speed: 253.29 [samples/sec]
2020-08-27 13:45:00,706: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:45:00,706: [worker  0] Epoch: 92
2020-08-27 13:45:00,706: [worker  0] Learning rate: 1.220703125e-07
2020-08-27 13:45:00,706: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:48:12,569: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:48:12,570: [worker  0] Eval  Speed: 224.19 [samples/sec]
2020-08-27 13:48:12,840: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:48:12,840: [worker  0] Epoch: 93
2020-08-27 13:48:12,840: [worker  0] Learning rate: 1.220703125e-07
2020-08-27 13:48:12,840: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:51:23,742: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:51:23,742: [worker  0] Eval  Speed: 253.44 [samples/sec]
2020-08-27 13:51:24,025: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:51:24,025: [worker  0] Epoch: 94
2020-08-27 13:51:24,025: [worker  0] Learning rate: 1.220703125e-07
2020-08-27 13:51:24,026: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:54:34,531: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:54:34,531: [worker  0] Eval  Speed: 250.05 [samples/sec]
2020-08-27 13:54:34,819: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:54:34,819: [worker  0] Epoch: 95
2020-08-27 13:54:34,819: [worker  0] Learning rate: 6.103515625e-08
2020-08-27 13:54:34,819: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:57:45,466: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 13:57:45,466: [worker  0] Eval  Speed: 253.77 [samples/sec]
2020-08-27 13:57:45,732: [worker  0] ----------------------------------------------------------------------
2020-08-27 13:57:45,732: [worker  0] Epoch: 96
2020-08-27 13:57:45,732: [worker  0] Learning rate: 6.103515625e-08
2020-08-27 13:57:45,732: [worker  0] ----------------------------------------------------------------------
2020-08-27 14:00:55,984: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 14:00:55,984: [worker  0] Eval  Speed: 253.06 [samples/sec]
2020-08-27 14:00:56,307: [worker  0] ----------------------------------------------------------------------
2020-08-27 14:00:56,307: [worker  0] Epoch: 97
2020-08-27 14:00:56,307: [worker  0] Learning rate: 6.103515625e-08
2020-08-27 14:00:56,307: [worker  0] ----------------------------------------------------------------------
2020-08-27 14:04:06,925: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 14:04:06,925: [worker  0] Eval  Speed: 251.73 [samples/sec]
2020-08-27 14:04:07,220: [worker  0] ----------------------------------------------------------------------
2020-08-27 14:04:07,221: [worker  0] Epoch: 98
2020-08-27 14:04:07,221: [worker  0] Learning rate: 6.103515625e-08
2020-08-27 14:04:07,221: [worker  0] ----------------------------------------------------------------------
2020-08-27 14:07:18,147: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 14:07:18,147: [worker  0] Eval  Speed: 251.78 [samples/sec]
2020-08-27 14:07:18,440: [worker  0] ----------------------------------------------------------------------
2020-08-27 14:07:18,440: [worker  0] Epoch: 99
2020-08-27 14:07:18,440: [worker  0] Learning rate: 3.0517578125e-08
2020-08-27 14:07:18,440: [worker  0] ----------------------------------------------------------------------
2020-08-27 14:10:29,650: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 14:10:29,650: [worker  0] Eval  Speed: 250.08 [samples/sec]
2020-08-27 14:10:30,109: [worker  0] ----------------------------------------------------------------------
2020-08-27 14:10:30,109: [worker  0] Epoch: 100
2020-08-27 14:10:30,109: [worker  0] Learning rate: 3.0517578125e-08
2020-08-27 14:10:30,109: [worker  0] ----------------------------------------------------------------------
2020-08-27 14:13:40,549: [worker  0] Eval   Loss: -1.105e+01
2020-08-27 14:13:40,549: [worker  0] Eval  Speed: 253.91 [samples/sec]

[vincentqb]

Training on an open dataset like LibriMix would allow for more users to experiment with the dataset

[vincentqb]

FYI: Initial training result

Best evaluation SI_SNR: 11.22 (Epoch: 54)

Some separation result (from top to bottom, mixed signal, source 1, source 2, separated speech 1, separated speech 2)

What metric are you using to know how good this value is?

[vincentqb]

log

Looking at the log, do you have a suggested approach to parse the output and generate plots for training/etc say per epoch or iterations?

[mthrok]

@mpariente @sw005320

In your frameworks, did you get similar number as mentioned in the paper?

I am getting the following numbers
https://github.com/mthrok/audio/tree/conv-tasnet/examples/source_separation/conv_tasnet#evaluation
I am not sure which exact type of WSJ0-mix dataset the paper (or in general) follows. (the total hours of the dataset described in the paper does not match with what I generated.)

The configuration I did follows the paper https://github.com/mthrok/audio/tree/conv-tasnet/examples/source_separation/conv_tasnet#default-training-configurations

I think my result is good enough for the sake of example, and do not intend to pursue more in the direction of reproducibility but if you have some insights, I am happy to listen.

[mpariente]

In your frameworks, did you get similar number as mentioned in the paper?

The paper reports 15.2 dB improvement, we get 16.2dB with our best ConvTasNet model.

I am not sure which exact type of WSJ0-mix dataset the paper (or in general) follows.

The only version, which is generated with the file available on this page.

the total hours of the dataset described in the paper does not match with what I generated.

What are those numbers in your case?

The configuration I did follows the paper

• Only the first 4sec are used for each examples right? I think you will improve if you select random 4s segments instead.
• I don't see a mention of the learning rate decay, did you implement it? It helps a bit in my experience.
• Did you compute the metrics after saving/reloading? SI-SNR loss usually produces large outputs that will clip, and it will reduce the overall metrics if they are computed after saving/reloading.

[mthrok]

@mpariente

Thanks for the reply.

The paper reports 15.2 dB improvement, we get 16.2dB with our best ConvTasNet model.

Nice 👍

The only version, which is generated with the file available on this page.

I mean, there are min and max mixture strategies. I am using them separately, but should I use both of them?

What are those numbers in your case?

The following is the directories and total audio files / durations I get when I ran the MERL script. Conv-TasNet paper says the following but none of the directory has the corresponding total hours.

We evaluated our system on two-speaker and three-speaker speech separation problems using the WSJ0-2mix and WSJ0- 3mix datasets [25]. 30 hours of training and 10 hours of validation data are generated from speakers in si tr s from the datasets.

wsj-mix dataset

wsj0-mix
├── 2speakers
│   ├── wav16k
│   │   ├── max
│   │   │   ├── cv - 11:53:36.14 ( 5000) - from si_tr_s
│   │   │   ├── tr - 46:56:16.34 (20000) - from si_tr_s
│   │   │   └── tt - 07:20:48.49 ( 3000) - from si_dt_05 and si_et_05
│   │   └── min
│   │       ├── cv - 07:40:22.00 ( 5000)
│   │       ├── tr - 30:22:49.09 (20000)
│   │       └── tt - 04:49:33.23 ( 3000)
│   └── wav8k
│       ├── max
│       │   ├── cv - 11:53:36.26 ( 5000)
│       │   ├── tr - 46:56:16.81 (20000)
│       │   └── tt - 07:20:48.57 ( 3000)
│       └── min
│           ├── cv - 07:40:22.10 ( 5000)
│           ├── tr - 30:22:49.52 (20000)
│           └── tt - 04:49:33.31 ( 3000)
└── 3speakers
    ├── wav16k
    │   ├── max
    │   │   ├── cv - 13:05:48.92 ( 5000) - from si_tr_s
    │   │   ├── tr - 52:02:04.90 (20000) - from si_tr_s
    │   │   └── tt - 08:05:21.30 ( 3000) - from si_dt_05 and si_et_05
    │   └── min
    │       ├── cv - 06:52:25.17 ( 5000)
    │       ├── tr - 27:10:19.31 (20000)
    │       └── tt - 04:21:03.82 ( 3000)
    └── wav8k
        ├── max
        │   ├── cv - 13:05:49.04 ( 5000)
        │   ├── tr - 52:02:05.37 (20000)
        │   └── tt - 08:05:21.38 ( 3000)
        └── min
            ├── cv - 06:52:25.27 ( 5000)
            ├── tr - 27:10:19.74 (20000)
            └── tt - 04:21:03.90 ( 3000)

• Only the first 4sec are used for each examples right? I think you will improve if you select random 4s segments instead.

Yes, I can try the random segmentation.

• I don't see a mention of the learning rate decay, did you implement it? It helps a bit in my experience.

I halved leaning rate when validation accuracy does not improve for 3 epochs, like here.

• Did you compute the metrics after saving/reloading? SI-SNR loss usually produces large outputs that will clip, and it will reduce the overall metrics if they are computed after saving/reloading.

I observed the same large outputs. Good to know that it's coming from SI-SNR. I computed the metrics on the fly in-memory. So clipping should not interfere.

[mpariente]

• min and max are the same mixtures, the min truncates the mixture when the one of both utterance has ended, max waits until both ended. We usually train with min only.

• In the min version, there is 30h of training, 7h of validation.. this is looks right. By the way, the output you generated is super cool ! Would you mind sharing the command for that? That would be amazing! 🛵

• For the rest, everything seems fine on paper. Just FYI, our best model used batch size 12 over 4 GPUs, with segments of 3sec.

[mthrok]

@mpariente

Thanks for the quick reply.

• min and max are the same mixtures, the min truncates the mixture when the one of both utterance has ended, max waits until both ended. We usually train with min only.

Thanks, this is very informative and a good learning opportunity..

• In the min version, there is 30h of training, 7h of validation.. this is looks right. By the way, the output you generated is super cool ! Would you mind sharing the command for that? That would be amazing! 🛵

The output is a manual combination of tree and soxi -Td commands.

• For the rest, everything seems fine on paper. Just FYI, our best model used batch size 12 over 4 GPUs, with segments of 3sec.

Interesting. Now that I realize that the paper has no mention of the batch size and I blindly used 32. I will try 12 too.