Upload folder using huggingface_hub
Browse files
clipcls_vit_b16_s512m_bs16k_mix0_0/checkpoints/epoch_4.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:68f0f612e2c98e07583d1446a03c9986d8b8986e45fc72b9d49389ce5d023679
|
| 3 |
+
size 2252180864
|
clipcls_vit_b16_s512m_bs16k_mix0_0/out.log
ADDED
|
@@ -0,0 +1,589 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2025-05-06,21:28:36 | INFO | No latest resume checkpoint found in ./logs-lr1e-3-datacomp/clipcls_vit_b16_s512m_bs16k_mix0_0/checkpoints.
|
| 2 |
+
2025-05-06,21:28:37 | INFO | Running in distributed mode with multiple processes. Device: cuda:0.Process (global: 0, local 0), total 16.
|
| 3 |
+
2025-05-06,21:28:37 | INFO | Loaded CLIPCLS-ViT-B-16 model config.
|
| 4 |
+
2025-05-06,21:28:40 | INFO | Model:
|
| 5 |
+
2025-05-06,21:28:40 | INFO | CLIPCLS(
|
| 6 |
+
(visual): VisionTransformer(
|
| 7 |
+
(conv1): Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16), bias=False)
|
| 8 |
+
(patch_dropout): Identity()
|
| 9 |
+
(ln_pre): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
|
| 10 |
+
(transformer): Transformer(
|
| 11 |
+
(resblocks): ModuleList(
|
| 12 |
+
(0-11): 12 x ResidualAttentionBlock(
|
| 13 |
+
(ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
|
| 14 |
+
(attn): MultiheadAttention(
|
| 15 |
+
(out_proj): NonDynamicallyQuantizableLinear(in_features=768, out_features=768, bias=True)
|
| 16 |
+
)
|
| 17 |
+
(ls_1): Identity()
|
| 18 |
+
(ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
|
| 19 |
+
(mlp): Sequential(
|
| 20 |
+
(c_fc): Linear(in_features=768, out_features=3072, bias=True)
|
| 21 |
+
(gelu): GELU(approximate='none')
|
| 22 |
+
(c_proj): Linear(in_features=3072, out_features=768, bias=True)
|
| 23 |
+
)
|
| 24 |
+
(ls_2): Identity()
|
| 25 |
+
)
|
| 26 |
+
)
|
| 27 |
+
)
|
| 28 |
+
(ln_post): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
|
| 29 |
+
)
|
| 30 |
+
(text): TextTransformer(
|
| 31 |
+
(token_embedding): Embedding(49408, 512)
|
| 32 |
+
(transformer): Transformer(
|
| 33 |
+
(resblocks): ModuleList(
|
| 34 |
+
(0-11): 12 x ResidualAttentionBlock(
|
| 35 |
+
(ln_1): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
|
| 36 |
+
(attn): MultiheadAttention(
|
| 37 |
+
(out_proj): NonDynamicallyQuantizableLinear(in_features=512, out_features=512, bias=True)
|
| 38 |
+
)
|
| 39 |
+
(ls_1): Identity()
|
| 40 |
+
(ln_2): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
|
| 41 |
+
(mlp): Sequential(
|
| 42 |
+
(c_fc): Linear(in_features=512, out_features=2048, bias=True)
|
| 43 |
+
(gelu): GELU(approximate='none')
|
| 44 |
+
(c_proj): Linear(in_features=2048, out_features=512, bias=True)
|
| 45 |
+
)
|
| 46 |
+
(ls_2): Identity()
|
| 47 |
+
)
|
| 48 |
+
)
|
| 49 |
+
)
|
| 50 |
+
(ln_final): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
|
| 51 |
+
)
|
| 52 |
+
(text_decoder): MixClsHead(
|
| 53 |
+
(mlps): ModuleList()
|
| 54 |
+
(ln_mlp): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
|
| 55 |
+
(text_projection): Linear(in_features=768, out_features=49408, bias=True)
|
| 56 |
+
)
|
| 57 |
+
)
|
| 58 |
+
2025-05-06,21:28:40 | INFO | Params:
|
| 59 |
+
2025-05-06,21:28:40 | INFO | NDR_patch_size: 16
|
| 60 |
+
2025-05-06,21:28:40 | INFO | accum_freq: 1
|
| 61 |
+
2025-05-06,21:28:40 | INFO | aug_cfg: {}
|
| 62 |
+
2025-05-06,21:28:40 | INFO | batch_size: 1024
|
| 63 |
+
2025-05-06,21:28:40 | INFO | beta1: 0.9
|
| 64 |
+
2025-05-06,21:28:40 | INFO | beta2: 0.98
|
| 65 |
+
2025-05-06,21:28:40 | INFO | checkpoint_path: ./logs-lr1e-3-datacomp/clipcls_vit_b16_s512m_bs16k_mix0_0/checkpoints
|
| 66 |
+
2025-05-06,21:28:40 | INFO | coca_caption_loss_weight: 2.0
|
| 67 |
+
2025-05-06,21:28:40 | INFO | coca_contrastive_loss_weight: 1.0
|
| 68 |
+
2025-05-06,21:28:40 | INFO | copy_codebase: False
|
| 69 |
+
2025-05-06,21:28:40 | INFO | csv_caption_key: title
|
| 70 |
+
2025-05-06,21:28:40 | INFO | csv_img_key: filepath
|
| 71 |
+
2025-05-06,21:28:40 | INFO | csv_separator:
|
| 72 |
+
2025-05-06,21:28:40 | INFO | dataset_resampled: False
|
| 73 |
+
2025-05-06,21:28:40 | INFO | dataset_type: webdataset
|
| 74 |
+
2025-05-06,21:28:40 | INFO | ddp_static_graph: True
|
| 75 |
+
2025-05-06,21:28:40 | INFO | debug: False
|
| 76 |
+
2025-05-06,21:28:40 | INFO | delete_prev_step_ckpt: True
|
| 77 |
+
2025-05-06,21:28:40 | INFO | delete_previous_checkpoint: False
|
| 78 |
+
2025-05-06,21:28:40 | INFO | device: cuda:0
|
| 79 |
+
2025-05-06,21:28:40 | INFO | dist_backend: nccl
|
| 80 |
+
2025-05-06,21:28:40 | INFO | dist_url: env://
|
| 81 |
+
2025-05-06,21:28:40 | INFO | distill: False
|
| 82 |
+
2025-05-06,21:28:40 | INFO | distill_model: None
|
| 83 |
+
2025-05-06,21:28:40 | INFO | distill_pretrained: None
|
| 84 |
+
2025-05-06,21:28:40 | INFO | distributed: True
|
| 85 |
+
2025-05-06,21:28:40 | INFO | epochs: 4
|
| 86 |
+
2025-05-06,21:28:40 | INFO | epochs_cooldown: None
|
| 87 |
+
2025-05-06,21:28:40 | INFO | eps: 1e-06
|
| 88 |
+
2025-05-06,21:28:40 | INFO | force_custom_text: False
|
| 89 |
+
2025-05-06,21:28:40 | INFO | force_image_size: 224
|
| 90 |
+
2025-05-06,21:28:40 | INFO | force_patch_dropout: None
|
| 91 |
+
2025-05-06,21:28:40 | INFO | force_quick_gelu: False
|
| 92 |
+
2025-05-06,21:28:40 | INFO | gather_with_grad: True
|
| 93 |
+
2025-05-06,21:28:40 | INFO | global_batch_size: 16384
|
| 94 |
+
2025-05-06,21:28:40 | INFO | grad_checkpointing: True
|
| 95 |
+
2025-05-06,21:28:40 | INFO | grad_clip_norm: None
|
| 96 |
+
2025-05-06,21:28:40 | INFO | horovod: False
|
| 97 |
+
2025-05-06,21:28:40 | INFO | image_interpolation: None
|
| 98 |
+
2025-05-06,21:28:40 | INFO | image_mean: None
|
| 99 |
+
2025-05-06,21:28:40 | INFO | image_resize_mode: None
|
| 100 |
+
2025-05-06,21:28:40 | INFO | image_std: None
|
| 101 |
+
2025-05-06,21:28:40 | INFO | imagenet_v2: None
|
| 102 |
+
2025-05-06,21:28:40 | INFO | imagenet_val: /mnt/bn/zilongdata-hl/dataset/imagenet/val
|
| 103 |
+
2025-05-06,21:28:40 | INFO | is_cls_token: True
|
| 104 |
+
2025-05-06,21:28:40 | INFO | local_loss: True
|
| 105 |
+
2025-05-06,21:28:40 | INFO | local_rank: 0
|
| 106 |
+
2025-05-06,21:28:40 | INFO | lock_image: False
|
| 107 |
+
2025-05-06,21:28:40 | INFO | lock_image_freeze_bn_stats: False
|
| 108 |
+
2025-05-06,21:28:40 | INFO | lock_image_unlocked_groups: 0
|
| 109 |
+
2025-05-06,21:28:40 | INFO | lock_text: False
|
| 110 |
+
2025-05-06,21:28:40 | INFO | lock_text_freeze_layer_norm: False
|
| 111 |
+
2025-05-06,21:28:40 | INFO | lock_text_unlocked_layers: 0
|
| 112 |
+
2025-05-06,21:28:40 | INFO | log_every_n_steps: 128
|
| 113 |
+
2025-05-06,21:28:40 | INFO | log_level: 20
|
| 114 |
+
2025-05-06,21:28:40 | INFO | log_local: False
|
| 115 |
+
2025-05-06,21:28:40 | INFO | log_path: ./logs-lr1e-3-datacomp/clipcls_vit_b16_s512m_bs16k_mix0_0/out.log
|
| 116 |
+
2025-05-06,21:28:40 | INFO | logs: ./logs-lr1e-3-datacomp
|
| 117 |
+
2025-05-06,21:28:40 | INFO | lr: 0.001
|
| 118 |
+
2025-05-06,21:28:40 | INFO | lr_cooldown_end: 0.0
|
| 119 |
+
2025-05-06,21:28:40 | INFO | lr_cooldown_power: 1.0
|
| 120 |
+
2025-05-06,21:28:40 | INFO | lr_scheduler: cosine
|
| 121 |
+
2025-05-06,21:28:40 | INFO | max_seq_len: 15000
|
| 122 |
+
2025-05-06,21:28:40 | INFO | model: CLIPCLS-ViT-B-16
|
| 123 |
+
2025-05-06,21:28:40 | INFO | name: clipcls_vit_b16_s512m_bs16k_mix0_0
|
| 124 |
+
2025-05-06,21:28:40 | INFO | native_dynamic_resolution: False
|
| 125 |
+
2025-05-06,21:28:40 | INFO | no_set_device_rank: False
|
| 126 |
+
2025-05-06,21:28:40 | INFO | only_packing: False
|
| 127 |
+
2025-05-06,21:28:40 | INFO | precision: amp
|
| 128 |
+
2025-05-06,21:28:40 | INFO | pretrained:
|
| 129 |
+
2025-05-06,21:28:40 | INFO | pretrained_image:
|
| 130 |
+
2025-05-06,21:28:40 | INFO | pretrained_text:
|
| 131 |
+
2025-05-06,21:28:40 | INFO | rank: 0
|
| 132 |
+
2025-05-06,21:28:40 | INFO | remote_sync: None
|
| 133 |
+
2025-05-06,21:28:40 | INFO | remote_sync_frequency: 300
|
| 134 |
+
2025-05-06,21:28:40 | INFO | remote_sync_protocol: s3
|
| 135 |
+
2025-05-06,21:28:40 | INFO | report_to: wandb
|
| 136 |
+
2025-05-06,21:28:40 | INFO | resume: None
|
| 137 |
+
2025-05-06,21:28:40 | INFO | rope_attn_num_heads: 12
|
| 138 |
+
2025-05-06,21:28:40 | INFO | rope_model_width: 768
|
| 139 |
+
2025-05-06,21:28:40 | INFO | save_every_n_steps: 6104
|
| 140 |
+
2025-05-06,21:28:40 | INFO | save_frequency: 1
|
| 141 |
+
2025-05-06,21:28:40 | INFO | save_most_recent: False
|
| 142 |
+
2025-05-06,21:28:40 | INFO | seed: 0
|
| 143 |
+
2025-05-06,21:28:40 | INFO | siglip: False
|
| 144 |
+
2025-05-06,21:28:40 | INFO | skip_scheduler: False
|
| 145 |
+
2025-05-06,21:28:40 | INFO | tensorboard: False
|
| 146 |
+
2025-05-06,21:28:40 | INFO | tensorboard_path:
|
| 147 |
+
2025-05-06,21:28:40 | INFO | torchcompile: False
|
| 148 |
+
2025-05-06,21:28:40 | INFO | torchscript: False
|
| 149 |
+
2025-05-06,21:28:40 | INFO | trace: False
|
| 150 |
+
2025-05-06,21:28:40 | INFO | train_data: /mnt/bn/zilongdata-hl/dataset/Recap-DataComp-1B-Dataset/{000000..140146}.tar
|
| 151 |
+
2025-05-06,21:28:40 | INFO | train_data_upsampling_factors: None
|
| 152 |
+
2025-05-06,21:28:40 | INFO | train_num_samples: 128000000
|
| 153 |
+
2025-05-06,21:28:40 | INFO | use_bn_sync: False
|
| 154 |
+
2025-05-06,21:28:40 | INFO | use_bnb_linear: None
|
| 155 |
+
2025-05-06,21:28:40 | INFO | val_data: None
|
| 156 |
+
2025-05-06,21:28:40 | INFO | val_frequency: 1
|
| 157 |
+
2025-05-06,21:28:40 | INFO | val_num_samples: None
|
| 158 |
+
2025-05-06,21:28:40 | INFO | val_steps: 0
|
| 159 |
+
2025-05-06,21:28:40 | INFO | wandb: True
|
| 160 |
+
2025-05-06,21:28:40 | INFO | wandb_notes:
|
| 161 |
+
2025-05-06,21:28:40 | INFO | wandb_project_name: cls-clip-NDR
|
| 162 |
+
2025-05-06,21:28:40 | INFO | warmup: 500
|
| 163 |
+
2025-05-06,21:28:40 | INFO | wd: 0.2
|
| 164 |
+
2025-05-06,21:28:40 | INFO | workers: 1
|
| 165 |
+
2025-05-06,21:28:40 | INFO | world_size: 16
|
| 166 |
+
2025-05-06,21:28:40 | INFO | zeroshot_frequency: 4
|
| 167 |
+
2025-05-06,21:28:40 | INFO | zeroshot_steps: 0
|
| 168 |
+
2025-05-06,21:28:57 | INFO | Start epoch 0
|
| 169 |
+
2025-05-06,21:29:12 | INFO | Train Epoch: 0 [ 16384/128008192 (0%)] Data (t): 8.003 Batch (t): 15.342, 1067.95/s, 66.7468/s/gpu LR: 0.000002 Logit Scale: 14.286 Class_loss: 11.336 (11.336) Contrastive_loss: 9.7933 (9.7933) Loss: 21.129 (21.129)
|
| 170 |
+
2025-05-06,21:30:59 | WARNING | Handling webdataset error (OSError('image file is truncated (44 bytes not processed)')). Ignoring.
|
| 171 |
+
2025-05-06,21:34:53 | WARNING | Handling webdataset error (OSError('image file is truncated (47 bytes not processed)')). Ignoring.
|
| 172 |
+
2025-05-06,21:41:08 | INFO | Train Epoch: 0 [ 2113536/128008192 (2%)] Data (t): 0.393 Batch (t): 5.591, 3000.01/s, 187.500/s/gpu LR: 0.000258 Logit Scale: 14.322 Class_loss: 7.6241 (9.4801) Contrastive_loss: 7.7843 (8.7888) Loss: 15.408 (18.269)
|
| 173 |
+
2025-05-06,21:46:20 | WARNING | Handling webdataset error (OSError('image file is truncated (68 bytes not processed)')). Ignoring.
|
| 174 |
+
2025-05-06,21:53:05 | INFO | Train Epoch: 0 [ 4210688/128008192 (3%)] Data (t): 0.380 Batch (t): 5.600, 2953.32/s, 184.582/s/gpu LR: 0.000514 Logit Scale: 14.560 Class_loss: 7.5209 (8.8270) Contrastive_loss: 7.3107 (8.2961) Loss: 14.832 (17.123)
|
| 175 |
+
2025-05-06,21:55:26 | WARNING | Handling webdataset error (OSError('image file is truncated (25 bytes not processed)')). Ignoring.
|
| 176 |
+
2025-05-06,22:05:06 | INFO | Train Epoch: 0 [ 6307840/128008192 (5%)] Data (t): 0.377 Batch (t): 5.639, 2974.43/s, 185.902/s/gpu LR: 0.000770 Logit Scale: 15.116 Class_loss: 7.3909 (8.4680) Contrastive_loss: 6.9805 (7.9672) Loss: 14.371 (16.435)
|
| 177 |
+
2025-05-06,22:16:54 | INFO | Train Epoch: 0 [ 8404992/128008192 (7%)] Data (t): 0.361 Batch (t): 5.527, 2829.16/s, 176.822/s/gpu LR: 0.001000 Logit Scale: 16.589 Class_loss: 7.2188 (8.2182) Contrastive_loss: 6.9215 (7.7580) Loss: 14.140 (15.976)
|
| 178 |
+
2025-05-06,22:28:45 | INFO | Train Epoch: 0 [ 10502144/128008192 (8%)] Data (t): 0.371 Batch (t): 5.555, 2874.45/s, 179.653/s/gpu LR: 0.001000 Logit Scale: 18.327 Class_loss: 7.0084 (8.0165) Contrastive_loss: 5.5007 (7.3818) Loss: 12.509 (15.398)
|
| 179 |
+
2025-05-06,22:31:34 | WARNING | Handling webdataset error (OSError('image file is truncated (104 bytes not processed)')). Ignoring.
|
| 180 |
+
2025-05-06,22:31:42 | WARNING | Handling webdataset error (OSError('image file is truncated (21 bytes not processed)')). Ignoring.
|
| 181 |
+
2025-05-06,22:40:38 | INFO | Train Epoch: 0 [ 12599296/128008192 (10%)] Data (t): 0.377 Batch (t): 5.568, 2949.88/s, 184.368/s/gpu LR: 0.001000 Logit Scale: 20.775 Class_loss: 6.7439 (7.8347) Contrastive_loss: 4.7691 (7.0086) Loss: 11.513 (14.843)
|
| 182 |
+
2025-05-06,22:52:31 | INFO | Train Epoch: 0 [ 14696448/128008192 (11%)] Data (t): 0.378 Batch (t): 5.573, 2975.66/s, 185.978/s/gpu LR: 0.001000 Logit Scale: 23.459 Class_loss: 6.7193 (7.6953) Contrastive_loss: 3.9783 (6.6298) Loss: 10.698 (14.325)
|
| 183 |
+
2025-05-06,22:55:33 | WARNING | Handling webdataset error (OSError('image file is truncated (32 bytes not processed)')). Ignoring.
|
| 184 |
+
2025-05-06,23:01:53 | WARNING | Handling webdataset error (OSError('image file is truncated (23 bytes not processed)')). Ignoring.
|
| 185 |
+
2025-05-06,23:04:26 | INFO | Train Epoch: 0 [ 16793600/128008192 (13%)] Data (t): 0.381 Batch (t): 5.590, 2901.18/s, 181.324/s/gpu LR: 0.000999 Logit Scale: 26.398 Class_loss: 6.5582 (7.5689) Contrastive_loss: 3.5441 (6.2869) Loss: 10.102 (13.856)
|
| 186 |
+
2025-05-06,23:16:25 | INFO | Train Epoch: 0 [ 18890752/128008192 (15%)] Data (t): 0.383 Batch (t): 5.611, 2892.89/s, 180.806/s/gpu LR: 0.000999 Logit Scale: 29.261 Class_loss: 6.4753 (7.4596) Contrastive_loss: 2.8936 (5.9476) Loss: 9.3688 (13.407)
|
| 187 |
+
2025-05-06,23:28:22 | INFO | Train Epoch: 0 [ 20987904/128008192 (16%)] Data (t): 0.379 Batch (t): 5.608, 2883.07/s, 180.192/s/gpu LR: 0.000998 Logit Scale: 32.592 Class_loss: 6.2939 (7.3536) Contrastive_loss: 2.7761 (5.6593) Loss: 9.0700 (13.013)
|
| 188 |
+
2025-05-06,23:40:20 | INFO | Train Epoch: 0 [ 23085056/128008192 (18%)] Data (t): 0.386 Batch (t): 5.602, 2952.01/s, 184.501/s/gpu LR: 0.000998 Logit Scale: 35.558 Class_loss: 6.3114 (7.2668) Contrastive_loss: 2.2600 (5.3760) Loss: 8.5714 (12.643)
|
| 189 |
+
2025-05-06,23:52:19 | INFO | Train Epoch: 0 [ 25182208/128008192 (20%)] Data (t): 0.375 Batch (t): 5.619, 2943.33/s, 183.958/s/gpu LR: 0.000997 Logit Scale: 39.130 Class_loss: 6.2089 (7.1854) Contrastive_loss: 2.2930 (5.1389) Loss: 8.5019 (12.324)
|
| 190 |
+
2025-05-07,00:04:16 | INFO | Train Epoch: 0 [ 27279360/128008192 (21%)] Data (t): 0.380 Batch (t): 5.606, 2947.47/s, 184.217/s/gpu LR: 0.000996 Logit Scale: 42.133 Class_loss: 6.2164 (7.1162) Contrastive_loss: 2.2540 (4.9328) Loss: 8.4704 (12.049)
|
| 191 |
+
2025-05-07,00:05:43 | WARNING | Handling webdataset error (OSError('image file is truncated (59 bytes not processed)')). Ignoring.
|
| 192 |
+
2025-05-07,00:16:17 | INFO | Train Epoch: 0 [ 29376512/128008192 (23%)] Data (t): 0.382 Batch (t): 5.633, 2942.06/s, 183.879/s/gpu LR: 0.000996 Logit Scale: 44.720 Class_loss: 6.2026 (7.0553) Contrastive_loss: 1.8163 (4.7250) Loss: 8.0188 (11.780)
|
| 193 |
+
2025-05-07,00:17:45 | WARNING | Handling webdataset error (OSError('image file is truncated (1 bytes not processed)')). Ignoring.
|
| 194 |
+
2025-05-07,00:28:18 | INFO | Train Epoch: 0 [ 31473664/128008192 (25%)] Data (t): 0.391 Batch (t): 5.627, 2892.51/s, 180.782/s/gpu LR: 0.000995 Logit Scale: 44.663 Class_loss: 6.2804 (7.0068) Contrastive_loss: 2.5745 (4.5906) Loss: 8.8549 (11.597)
|
| 195 |
+
2025-05-07,00:40:16 | INFO | Train Epoch: 0 [ 33570816/128008192 (26%)] Data (t): 0.371 Batch (t): 5.610, 2915.08/s, 182.193/s/gpu LR: 0.000994 Logit Scale: 46.638 Class_loss: 6.0319 (6.9495) Contrastive_loss: 1.7493 (4.4235) Loss: 7.7812 (11.373)
|
| 196 |
+
2025-05-07,00:44:01 | WARNING | Handling webdataset error (OSError('image file is truncated (0 bytes not processed)')). Ignoring.
|
| 197 |
+
2025-05-07,00:52:13 | INFO | Train Epoch: 0 [ 35667968/128008192 (28%)] Data (t): 0.378 Batch (t): 5.601, 2945.21/s, 184.076/s/gpu LR: 0.000993 Logit Scale: 48.722 Class_loss: 6.0626 (6.9002) Contrastive_loss: 1.6700 (4.2705) Loss: 7.7326 (11.171)
|
| 198 |
+
2025-05-07,01:03:20 | WARNING | Handling webdataset error (OSError('image file is truncated (18 bytes not processed)')). Ignoring.
|
| 199 |
+
2025-05-07,01:04:14 | INFO | Train Epoch: 0 [ 37765120/128008192 (30%)] Data (t): 0.382 Batch (t): 5.634, 2894.76/s, 180.923/s/gpu LR: 0.000992 Logit Scale: 50.669 Class_loss: 6.0058 (6.8531) Contrastive_loss: 1.5479 (4.1272) Loss: 7.5537 (10.980)
|
| 200 |
+
2025-05-07,01:05:03 | WARNING | Handling webdataset error (OSError('image file is truncated (59 bytes not processed)')). Ignoring.
|
| 201 |
+
2025-05-07,01:14:56 | WARNING | Handling webdataset error (OSError('image file is truncated (84 bytes not processed)')). Ignoring.
|
| 202 |
+
2025-05-07,01:16:13 | INFO | Train Epoch: 0 [ 39862272/128008192 (31%)] Data (t): 0.384 Batch (t): 5.615, 2894.00/s, 180.875/s/gpu LR: 0.000990 Logit Scale: 52.166 Class_loss: 5.8499 (6.8030) Contrastive_loss: 1.5213 (3.9969) Loss: 7.3712 (10.800)
|
| 203 |
+
2025-05-07,01:28:07 | INFO | Train Epoch: 0 [ 41959424/128008192 (33%)] Data (t): 0.372 Batch (t): 5.585, 2942.10/s, 183.881/s/gpu LR: 0.000989 Logit Scale: 53.521 Class_loss: 5.8661 (6.7584) Contrastive_loss: 1.3359 (3.8702) Loss: 7.2020 (10.629)
|
| 204 |
+
2025-05-07,01:40:05 | INFO | Train Epoch: 0 [ 44056576/128008192 (34%)] Data (t): 0.380 Batch (t): 5.609, 2940.29/s, 183.768/s/gpu LR: 0.000988 Logit Scale: 54.761 Class_loss: 5.9538 (6.7218) Contrastive_loss: 1.4329 (3.7594) Loss: 7.3868 (10.481)
|
| 205 |
+
2025-05-07,01:42:51 | WARNING | Handling webdataset error (OSError('image file is truncated (49 bytes not processed)')). Ignoring.
|
| 206 |
+
2025-05-07,01:51:57 | INFO | Train Epoch: 0 [ 46153728/128008192 (36%)] Data (t): 0.382 Batch (t): 5.561, 2914.96/s, 182.185/s/gpu LR: 0.000986 Logit Scale: 55.696 Class_loss: 5.8080 (6.6821) Contrastive_loss: 1.2553 (3.6506) Loss: 7.0633 (10.333)
|
| 207 |
+
2025-05-07,02:03:56 | INFO | Train Epoch: 0 [ 48250880/128008192 (38%)] Data (t): 0.497 Batch (t): 5.619, 2763.76/s, 172.735/s/gpu LR: 0.000984 Logit Scale: 56.778 Class_loss: 5.8930 (6.6492) Contrastive_loss: 1.3178 (3.5534) Loss: 7.2108 (10.203)
|
| 208 |
+
2025-05-07,02:15:54 | INFO | Train Epoch: 0 [ 50348032/128008192 (39%)] Data (t): 0.373 Batch (t): 5.609, 2904.83/s, 181.552/s/gpu LR: 0.000983 Logit Scale: 56.814 Class_loss: 6.4390 (6.6408) Contrastive_loss: 2.9523 (3.5293) Loss: 9.3913 (10.170)
|
| 209 |
+
2025-05-07,02:27:53 | INFO | Train Epoch: 0 [ 52445184/128008192 (41%)] Data (t): 0.382 Batch (t): 5.617, 2898.75/s, 181.172/s/gpu LR: 0.000981 Logit Scale: 57.396 Class_loss: 5.7772 (6.6076) Contrastive_loss: 1.1993 (3.4397) Loss: 6.9764 (10.047)
|
| 210 |
+
2025-05-07,02:39:53 | INFO | Train Epoch: 0 [ 54542336/128008192 (43%)] Data (t): 0.379 Batch (t): 5.624, 2909.22/s, 181.826/s/gpu LR: 0.000979 Logit Scale: 58.607 Class_loss: 5.8220 (6.5785) Contrastive_loss: 1.0580 (3.3515) Loss: 6.8799 (9.9300)
|
| 211 |
+
2025-05-07,02:51:52 | INFO | Train Epoch: 0 [ 56639488/128008192 (44%)] Data (t): 0.382 Batch (t): 5.616, 2880.96/s, 180.060/s/gpu LR: 0.000977 Logit Scale: 58.259 Class_loss: 5.7708 (6.5496) Contrastive_loss: 1.3076 (3.2785) Loss: 7.0784 (9.8281)
|
| 212 |
+
2025-05-07,03:03:51 | INFO | Train Epoch: 0 [ 58736640/128008192 (46%)] Data (t): 0.378 Batch (t): 5.619, 2963.59/s, 185.224/s/gpu LR: 0.000975 Logit Scale: 59.852 Class_loss: 5.8159 (6.5243) Contrastive_loss: 1.0659 (3.2022) Loss: 6.8818 (9.7265)
|
| 213 |
+
2025-05-07,03:04:06 | WARNING | Handling webdataset error (OSError('image file is truncated (107 bytes not processed)')). Ignoring.
|
| 214 |
+
2025-05-07,03:04:26 | WARNING | Handling webdataset error (OSError('image file is truncated (82 bytes not processed)')). Ignoring.
|
| 215 |
+
2025-05-07,03:05:22 | WARNING | Handling webdataset error (OSError('image file is truncated (73 bytes not processed)')). Ignoring.
|
| 216 |
+
2025-05-07,03:15:52 | INFO | Train Epoch: 0 [ 60833792/128008192 (48%)] Data (t): 0.378 Batch (t): 5.629, 2889.85/s, 180.615/s/gpu LR: 0.000973 Logit Scale: 60.610 Class_loss: 5.8027 (6.5003) Contrastive_loss: 1.0169 (3.1293) Loss: 6.8197 (9.6296)
|
| 217 |
+
2025-05-07,03:27:52 | INFO | Train Epoch: 0 [ 62930944/128008192 (49%)] Data (t): 0.382 Batch (t): 5.626, 2870.34/s, 179.396/s/gpu LR: 0.000971 Logit Scale: 61.352 Class_loss: 5.6865 (6.4740) Contrastive_loss: 1.1602 (3.0658) Loss: 6.8467 (9.5398)
|
| 218 |
+
2025-05-07,03:28:30 | WARNING | Handling webdataset error (OSError('image file is truncated (2 bytes not processed)')). Ignoring.
|
| 219 |
+
2025-05-07,03:39:57 | INFO | Train Epoch: 0 [ 65028096/128008192 (51%)] Data (t): 0.390 Batch (t): 5.669, 3002.73/s, 187.671/s/gpu LR: 0.000969 Logit Scale: 61.946 Class_loss: 5.7872 (6.4526) Contrastive_loss: 1.1982 (3.0075) Loss: 6.9854 (9.4600)
|
| 220 |
+
2025-05-07,03:40:29 | WARNING | Handling webdataset error (OSError('image file is truncated (54 bytes not processed)')). Ignoring.
|
| 221 |
+
2025-05-07,03:52:03 | INFO | Train Epoch: 0 [ 67125248/128008192 (52%)] Data (t): 0.383 Batch (t): 5.671, 2919.61/s, 182.476/s/gpu LR: 0.000967 Logit Scale: 62.507 Class_loss: 5.7666 (6.4318) Contrastive_loss: 1.0899 (2.9494) Loss: 6.8565 (9.3811)
|
| 222 |
+
2025-05-07,03:59:38 | WARNING | Handling webdataset error (OSError('image file is truncated (46 bytes not processed)')). Ignoring.
|
| 223 |
+
2025-05-07,04:02:01 | WARNING | Handling webdataset error (OSError('image file is truncated (87 bytes not processed)')). Ignoring.
|
| 224 |
+
2025-05-07,04:04:00 | INFO | Train Epoch: 0 [ 69222400/128008192 (54%)] Data (t): 0.380 Batch (t): 5.601, 2871.62/s, 179.476/s/gpu LR: 0.000964 Logit Scale: 62.957 Class_loss: 5.6868 (6.4099) Contrastive_loss: 1.0900 (2.8947) Loss: 6.7768 (9.3045)
|
| 225 |
+
2025-05-07,04:07:47 | WARNING | Handling webdataset error (OSError('image file is truncated (31 bytes not processed)')). Ignoring.
|
| 226 |
+
2025-05-07,04:15:59 | INFO | Train Epoch: 0 [ 71319552/128008192 (56%)] Data (t): 0.369 Batch (t): 5.612, 2945.35/s, 184.084/s/gpu LR: 0.000962 Logit Scale: 63.670 Class_loss: 5.7615 (6.3913) Contrastive_loss: 1.1061 (2.8436) Loss: 6.8676 (9.2349)
|
| 227 |
+
2025-05-07,04:28:02 | INFO | Train Epoch: 0 [ 73416704/128008192 (57%)] Data (t): 0.374 Batch (t): 5.651, 2742.65/s, 171.415/s/gpu LR: 0.000959 Logit Scale: 64.111 Class_loss: 5.7089 (6.3724) Contrastive_loss: 1.0207 (2.7929) Loss: 6.7296 (9.1653)
|
| 228 |
+
2025-05-07,04:40:05 | INFO | Train Epoch: 0 [ 75513856/128008192 (59%)] Data (t): 0.383 Batch (t): 5.651, 2891.25/s, 180.703/s/gpu LR: 0.000957 Logit Scale: 64.476 Class_loss: 5.7105 (6.3545) Contrastive_loss: 1.0377 (2.7455) Loss: 6.7481 (9.1000)
|
| 229 |
+
2025-05-07,04:47:30 | WARNING | Handling webdataset error (OSError('image file is truncated (64 bytes not processed)')). Ignoring.
|
| 230 |
+
2025-05-07,04:52:08 | INFO | Train Epoch: 0 [ 77611008/128008192 (61%)] Data (t): 0.458 Batch (t): 5.645, 2897.32/s, 181.082/s/gpu LR: 0.000954 Logit Scale: 64.969 Class_loss: 5.7210 (6.3378) Contrastive_loss: 1.0404 (2.7006) Loss: 6.7614 (9.0384)
|
| 231 |
+
2025-05-07,05:04:08 | INFO | Train Epoch: 0 [ 79708160/128008192 (62%)] Data (t): 0.382 Batch (t): 5.624, 2843.25/s, 177.703/s/gpu LR: 0.000951 Logit Scale: 65.429 Class_loss: 5.7019 (6.3215) Contrastive_loss: 1.1663 (2.6613) Loss: 6.8682 (8.9828)
|
| 232 |
+
2025-05-07,05:16:08 | INFO | Train Epoch: 0 [ 81805312/128008192 (64%)] Data (t): 0.379 Batch (t): 5.629, 2953.87/s, 184.617/s/gpu LR: 0.000948 Logit Scale: 65.982 Class_loss: 5.6505 (6.3047) Contrastive_loss: 1.0749 (2.6216) Loss: 6.7254 (8.9264)
|
| 233 |
+
2025-05-07,05:28:11 | INFO | Train Epoch: 0 [ 83902464/128008192 (66%)] Data (t): 0.378 Batch (t): 5.643, 2876.02/s, 179.751/s/gpu LR: 0.000945 Logit Scale: 66.359 Class_loss: 5.6046 (6.2877) Contrastive_loss: 1.2043 (2.5871) Loss: 6.8090 (8.8747)
|
| 234 |
+
2025-05-07,05:40:13 | INFO | Train Epoch: 0 [ 85999616/128008192 (67%)] Data (t): 0.382 Batch (t): 5.646, 2873.80/s, 179.613/s/gpu LR: 0.000942 Logit Scale: 66.824 Class_loss: 5.6359 (6.2721) Contrastive_loss: 0.97777 (2.5487) Loss: 6.6137 (8.8209)
|
| 235 |
+
2025-05-07,05:41:46 | WARNING | Handling webdataset error (OSError('image file is truncated (54 bytes not processed)')). Ignoring.
|
| 236 |
+
2025-05-07,05:43:28 | WARNING | Handling webdataset error (OSError('image file is truncated (72 bytes not processed)')). Ignoring.
|
| 237 |
+
2025-05-07,05:52:14 | INFO | Train Epoch: 0 [ 88096768/128008192 (69%)] Data (t): 0.379 Batch (t): 5.634, 2946.65/s, 184.166/s/gpu LR: 0.000939 Logit Scale: 67.075 Class_loss: 5.5719 (6.2559) Contrastive_loss: 1.1391 (2.5160) Loss: 6.7109 (8.7718)
|
| 238 |
+
2025-05-07,06:04:17 | INFO | Train Epoch: 0 [ 90193920/128008192 (70%)] Data (t): 0.372 Batch (t): 5.644, 2912.90/s, 182.056/s/gpu LR: 0.000936 Logit Scale: 67.517 Class_loss: 5.7016 (6.2433) Contrastive_loss: 0.99611 (2.4814) Loss: 6.6977 (8.7247)
|
| 239 |
+
2025-05-07,06:16:15 | INFO | Train Epoch: 0 [ 92291072/128008192 (72%)] Data (t): 0.369 Batch (t): 5.608, 2906.43/s, 181.652/s/gpu LR: 0.000933 Logit Scale: 67.794 Class_loss: 5.6116 (6.2292) Contrastive_loss: 0.93265 (2.4470) Loss: 6.5442 (8.6762)
|
| 240 |
+
2025-05-07,06:28:11 | INFO | Train Epoch: 0 [ 94388224/128008192 (74%)] Data (t): 0.366 Batch (t): 5.594, 2883.39/s, 180.212/s/gpu LR: 0.000930 Logit Scale: 68.292 Class_loss: 5.5442 (6.2143) Contrastive_loss: 0.81843 (2.4116) Loss: 6.3627 (8.6259)
|
| 241 |
+
2025-05-07,06:40:11 | INFO | Train Epoch: 0 [ 96485376/128008192 (75%)] Data (t): 0.384 Batch (t): 5.626, 2869.83/s, 179.364/s/gpu LR: 0.000926 Logit Scale: 68.598 Class_loss: 5.5936 (6.2011) Contrastive_loss: 1.0528 (2.3827) Loss: 6.6464 (8.5838)
|
| 242 |
+
2025-05-07,06:52:13 | INFO | Train Epoch: 0 [ 98582528/128008192 (77%)] Data (t): 0.381 Batch (t): 5.642, 2936.04/s, 183.503/s/gpu LR: 0.000923 Logit Scale: 68.924 Class_loss: 5.6726 (6.1901) Contrastive_loss: 0.81779 (2.3501) Loss: 6.4904 (8.5402)
|
| 243 |
+
2025-05-07,07:04:18 | INFO | Train Epoch: 0 [100679680/128008192 (79%)] Data (t): 0.434 Batch (t): 5.662, 2940.35/s, 183.772/s/gpu LR: 0.000919 Logit Scale: 69.184 Class_loss: 5.6083 (6.1782) Contrastive_loss: 0.81943 (2.3188) Loss: 6.4277 (8.4971)
|
| 244 |
+
2025-05-07,07:16:17 | INFO | Train Epoch: 0 [102776832/128008192 (80%)] Data (t): 0.381 Batch (t): 5.617, 2932.18/s, 183.261/s/gpu LR: 0.000916 Logit Scale: 69.563 Class_loss: 5.6346 (6.1674) Contrastive_loss: 0.92397 (2.2909) Loss: 6.5586 (8.4583)
|
| 245 |
+
2025-05-07,07:22:28 | WARNING | Handling webdataset error (OSError('image file is truncated (92 bytes not processed)')). Ignoring.
|
| 246 |
+
2025-05-07,07:28:17 | INFO | Train Epoch: 0 [104873984/128008192 (82%)] Data (t): 0.382 Batch (t): 5.630, 2893.27/s, 180.830/s/gpu LR: 0.000912 Logit Scale: 69.919 Class_loss: 5.5162 (6.1546) Contrastive_loss: 1.0599 (2.2668) Loss: 6.5761 (8.4214)
|
| 247 |
+
2025-05-07,07:40:21 | INFO | Train Epoch: 0 [106971136/128008192 (84%)] Data (t): 0.384 Batch (t): 5.652, 2964.23/s, 185.264/s/gpu LR: 0.000908 Logit Scale: 70.087 Class_loss: 5.5453 (6.1429) Contrastive_loss: 0.90469 (2.2406) Loss: 6.4500 (8.3835)
|
| 248 |
+
2025-05-07,07:52:23 | INFO | Train Epoch: 0 [109068288/128008192 (85%)] Data (t): 0.385 Batch (t): 5.645, 2902.11/s, 181.382/s/gpu LR: 0.000904 Logit Scale: 70.396 Class_loss: 5.5594 (6.1319) Contrastive_loss: 0.86448 (2.2146) Loss: 6.4239 (8.3465)
|
| 249 |
+
2025-05-07,07:55:39 | WARNING | Handling webdataset error (OSError('image file is truncated (59 bytes not processed)')). Ignoring.
|
| 250 |
+
2025-05-07,08:04:24 | INFO | Train Epoch: 0 [111165440/128008192 (87%)] Data (t): 0.381 Batch (t): 5.633, 2947.48/s, 184.218/s/gpu LR: 0.000900 Logit Scale: 70.649 Class_loss: 5.4925 (6.1200) Contrastive_loss: 0.91354 (2.1906) Loss: 6.4061 (8.3106)
|
| 251 |
+
2025-05-07,08:16:26 | INFO | Train Epoch: 0 [113262592/128008192 (88%)] Data (t): 0.379 Batch (t): 5.639, 2919.82/s, 182.489/s/gpu LR: 0.000897 Logit Scale: 70.836 Class_loss: 5.4755 (6.1083) Contrastive_loss: 0.96445 (2.1683) Loss: 6.4399 (8.2766)
|
| 252 |
+
2025-05-07,08:25:35 | WARNING | Handling webdataset error (OSError('image file is truncated (101 bytes not processed)')). Ignoring.
|
| 253 |
+
2025-05-07,08:28:27 | INFO | Train Epoch: 0 [115359744/128008192 (90%)] Data (t): 0.384 Batch (t): 5.633, 2903.69/s, 181.481/s/gpu LR: 0.000892 Logit Scale: 71.088 Class_loss: 5.5619 (6.0986) Contrastive_loss: 0.79368 (2.1437) Loss: 6.3556 (8.2423)
|
| 254 |
+
2025-05-07,08:40:29 | INFO | Train Epoch: 0 [117456896/128008192 (92%)] Data (t): 0.380 Batch (t): 5.637, 2937.21/s, 183.576/s/gpu LR: 0.000888 Logit Scale: 71.581 Class_loss: 5.5424 (6.0888) Contrastive_loss: 0.85852 (2.1212) Loss: 6.4010 (8.2100)
|
| 255 |
+
2025-05-07,08:52:31 | INFO | Train Epoch: 0 [119554048/128008192 (93%)] Data (t): 0.382 Batch (t): 5.641, 2911.36/s, 181.960/s/gpu LR: 0.000884 Logit Scale: 71.630 Class_loss: 5.5137 (6.0789) Contrastive_loss: 0.82509 (2.0988) Loss: 6.3388 (8.1777)
|
| 256 |
+
2025-05-07,08:56:34 | WARNING | Handling webdataset error (OSError('image file is truncated (82 bytes not processed)')). Ignoring.
|
| 257 |
+
2025-05-07,09:01:56 | WARNING | Handling webdataset error (OSError('image file is truncated (37 bytes not processed)')). Ignoring.
|
| 258 |
+
2025-05-07,09:04:30 | INFO | Train Epoch: 0 [121651200/128008192 (95%)] Data (t): 0.379 Batch (t): 5.618, 2897.23/s, 181.077/s/gpu LR: 0.000880 Logit Scale: 71.146 Class_loss: 5.5025 (6.0691) Contrastive_loss: 0.94402 (2.0792) Loss: 6.4465 (8.1484)
|
| 259 |
+
2025-05-07,09:07:19 | WARNING | Handling webdataset error (OSError('image file is truncated (4 bytes not processed)')). Ignoring.
|
| 260 |
+
2025-05-07,09:11:13 | WARNING | Handling webdataset error (OSError('image file is truncated (88 bytes not processed)')). Ignoring.
|
| 261 |
+
2025-05-07,09:16:27 | INFO | Train Epoch: 0 [123748352/128008192 (97%)] Data (t): 0.376 Batch (t): 5.606, 3024.49/s, 189.031/s/gpu LR: 0.000876 Logit Scale: 71.799 Class_loss: 5.4901 (6.0595) Contrastive_loss: 0.93858 (2.0602) Loss: 6.4287 (8.1197)
|
| 262 |
+
2025-05-07,09:28:20 | INFO | Train Epoch: 0 [125845504/128008192 (98%)] Data (t): 0.359 Batch (t): 5.568, 3008.26/s, 188.016/s/gpu LR: 0.000871 Logit Scale: 72.146 Class_loss: 5.5417 (6.0510) Contrastive_loss: 0.96214 (2.0422) Loss: 6.5039 (8.0932)
|
| 263 |
+
2025-05-07,09:40:20 | INFO | Train Epoch: 0 [127942656/128008192 (100%)] Data (t): 0.380 Batch (t): 5.623, 2831.12/s, 176.945/s/gpu LR: 0.000867 Logit Scale: 72.225 Class_loss: 5.5142 (6.0423) Contrastive_loss: 0.92228 (2.0242) Loss: 6.4365 (8.0665)
|
| 264 |
+
2025-05-07,09:40:42 | INFO | Train Epoch: 0 [128008192/128008192 (100%)] Data (t): 0.376 Batch (t): 5.640, 2959.57/s, 184.973/s/gpu LR: 0.000867 Logit Scale: 72.242 Class_loss: 5.5730 (6.0349) Contrastive_loss: 0.77674 (2.0044) Loss: 6.3498 (8.0392)
|
| 265 |
+
2025-05-07,09:40:51 | INFO | Start epoch 1
|
| 266 |
+
2025-05-07,09:41:03 | INFO | Train Epoch: 1 [ 16384/128008192 (0%)] Data (t): 7.413 Batch (t): 11.856, 1381.86/s, 86.3665/s/gpu LR: 0.000867 Logit Scale: 72.247 Class_loss: 5.5469 (5.5469) Contrastive_loss: 0.85398 (0.85398) Loss: 6.4009 (6.4009)
|
| 267 |
+
2025-05-07,09:53:00 | INFO | Train Epoch: 1 [ 2113536/128008192 (2%)] Data (t): 0.402 Batch (t): 5.602, 2985.40/s, 186.587/s/gpu LR: 0.000862 Logit Scale: 72.653 Class_loss: 5.4722 (5.5095) Contrastive_loss: 0.87008 (0.86203) Loss: 6.3423 (6.3716)
|
| 268 |
+
2025-05-07,09:56:24 | WARNING | Handling webdataset error (OSError('image file is truncated (53 bytes not processed)')). Ignoring.
|
| 269 |
+
2025-05-07,09:56:36 | WARNING | Handling webdataset error (OSError('image file is truncated (50 bytes not processed)')). Ignoring.
|
| 270 |
+
2025-05-07,10:04:56 | INFO | Train Epoch: 1 [ 4210688/128008192 (3%)] Data (t): 0.372 Batch (t): 5.590, 2993.99/s, 187.124/s/gpu LR: 0.000858 Logit Scale: 72.934 Class_loss: 5.4744 (5.4978) Contrastive_loss: 0.85247 (0.85884) Loss: 6.3269 (6.3567)
|
| 271 |
+
2025-05-07,10:16:49 | INFO | Train Epoch: 1 [ 6307840/128008192 (5%)] Data (t): 0.378 Batch (t): 5.570, 3017.39/s, 188.587/s/gpu LR: 0.000853 Logit Scale: 73.131 Class_loss: 5.5172 (5.5027) Contrastive_loss: 0.87147 (0.86200) Loss: 6.3886 (6.3647)
|
| 272 |
+
2025-05-07,10:28:50 | INFO | Train Epoch: 1 [ 8404992/128008192 (7%)] Data (t): 0.350 Batch (t): 5.630, 2933.15/s, 183.322/s/gpu LR: 0.000849 Logit Scale: 73.464 Class_loss: 5.4517 (5.4925) Contrastive_loss: 0.84384 (0.85837) Loss: 6.2955 (6.3508)
|
| 273 |
+
2025-05-07,10:40:42 | INFO | Train Epoch: 1 [ 10502144/128008192 (8%)] Data (t): 0.373 Batch (t): 5.569, 2782.66/s, 173.917/s/gpu LR: 0.000844 Logit Scale: 73.640 Class_loss: 5.5044 (5.4945) Contrastive_loss: 0.85727 (0.85819) Loss: 6.3617 (6.3526)
|
| 274 |
+
2025-05-07,10:52:41 | INFO | Train Epoch: 1 [ 12599296/128008192 (10%)] Data (t): 0.379 Batch (t): 5.613, 2776.39/s, 173.524/s/gpu LR: 0.000839 Logit Scale: 73.907 Class_loss: 5.4526 (5.4885) Contrastive_loss: 0.77509 (0.84632) Loss: 6.2277 (6.3348)
|
| 275 |
+
2025-05-07,11:04:41 | INFO | Train Epoch: 1 [ 14696448/128008192 (11%)] Data (t): 0.379 Batch (t): 5.628, 2905.23/s, 181.577/s/gpu LR: 0.000834 Logit Scale: 74.013 Class_loss: 5.4300 (5.4812) Contrastive_loss: 0.91649 (0.85509) Loss: 6.3465 (6.3363)
|
| 276 |
+
2025-05-07,11:16:42 | INFO | Train Epoch: 1 [ 16793600/128008192 (13%)] Data (t): 0.386 Batch (t): 5.635, 2874.28/s, 179.642/s/gpu LR: 0.000829 Logit Scale: 74.093 Class_loss: 5.5599 (5.4899) Contrastive_loss: 0.79345 (0.84824) Loss: 6.3533 (6.3382)
|
| 277 |
+
2025-05-07,11:28:45 | INFO | Train Epoch: 1 [ 18890752/128008192 (15%)] Data (t): 0.388 Batch (t): 5.644, 2835.52/s, 177.220/s/gpu LR: 0.000824 Logit Scale: 74.099 Class_loss: 5.4816 (5.4891) Contrastive_loss: 0.85683 (0.84910) Loss: 6.3384 (6.3382)
|
| 278 |
+
2025-05-07,11:31:37 | WARNING | Handling webdataset error (OSError('image file is truncated (6 bytes not processed)')). Ignoring.
|
| 279 |
+
2025-05-07,11:40:44 | INFO | Train Epoch: 1 [ 20987904/128008192 (16%)] Data (t): 0.384 Batch (t): 5.618, 2965.44/s, 185.340/s/gpu LR: 0.000819 Logit Scale: 74.393 Class_loss: 5.5652 (5.4960) Contrastive_loss: 0.83379 (0.84771) Loss: 6.3990 (6.3437)
|
| 280 |
+
2025-05-07,11:42:46 | WARNING | Handling webdataset error (OSError('image file is truncated (32 bytes not processed)')). Ignoring.
|
| 281 |
+
2025-05-07,11:43:09 | WARNING | Handling webdataset error (OSError('image file is truncated (66 bytes not processed)')). Ignoring.
|
| 282 |
+
2025-05-07,11:52:42 | INFO | Train Epoch: 1 [ 23085056/128008192 (18%)] Data (t): 0.372 Batch (t): 5.609, 2894.67/s, 180.917/s/gpu LR: 0.000814 Logit Scale: 74.664 Class_loss: 5.5323 (5.4990) Contrastive_loss: 0.67135 (0.83301) Loss: 6.2036 (6.3320)
|
| 283 |
+
2025-05-07,12:03:34 | WARNING | Handling webdataset error (OSError('image file is truncated (4 bytes not processed)')). Ignoring.
|
| 284 |
+
2025-05-07,12:04:37 | INFO | Train Epoch: 1 [ 25182208/128008192 (20%)] Data (t): 0.372 Batch (t): 5.584, 2940.42/s, 183.776/s/gpu LR: 0.000809 Logit Scale: 74.850 Class_loss: 5.4817 (5.4977) Contrastive_loss: 0.84736 (0.83411) Loss: 6.3290 (6.3318)
|
| 285 |
+
2025-05-07,12:13:40 | WARNING | Handling webdataset error (OSError('image file is truncated (5 bytes not processed)')). Ignoring.
|
| 286 |
+
2025-05-07,12:16:32 | INFO | Train Epoch: 1 [ 27279360/128008192 (21%)] Data (t): 0.380 Batch (t): 5.589, 2939.35/s, 183.709/s/gpu LR: 0.000804 Logit Scale: 75.037 Class_loss: 5.4534 (5.4945) Contrastive_loss: 0.83143 (0.83392) Loss: 6.2848 (6.3285)
|
| 287 |
+
2025-05-07,12:16:38 | WARNING | Handling webdataset error (OSError('image file is truncated (5 bytes not processed)')). Ignoring.
|
| 288 |
+
2025-05-07,12:20:23 | WARNING | Handling webdataset error (OSError('image file is truncated (131 bytes not processed)')). Ignoring.
|
| 289 |
+
2025-05-07,12:28:31 | INFO | Train Epoch: 1 [ 29376512/128008192 (23%)] Data (t): 0.380 Batch (t): 5.614, 2921.48/s, 182.593/s/gpu LR: 0.000799 Logit Scale: 75.315 Class_loss: 5.4661 (5.4926) Contrastive_loss: 0.82027 (0.83301) Loss: 6.2864 (6.3257)
|
| 290 |
+
2025-05-07,12:29:36 | WARNING | Handling webdataset error (OSError('image file is truncated (15 bytes not processed)')). Ignoring.
|
| 291 |
+
2025-05-07,12:40:30 | INFO | Train Epoch: 1 [ 31473664/128008192 (25%)] Data (t): 0.384 Batch (t): 5.620, 2734.03/s, 170.877/s/gpu LR: 0.000794 Logit Scale: 75.534 Class_loss: 5.4048 (5.4872) Contrastive_loss: 0.89120 (0.83665) Loss: 6.2960 (6.3238)
|
| 292 |
+
2025-05-07,12:52:29 | INFO | Train Epoch: 1 [ 33570816/128008192 (26%)] Data (t): 0.379 Batch (t): 5.615, 2944.62/s, 184.039/s/gpu LR: 0.000788 Logit Scale: 75.673 Class_loss: 5.3960 (5.4818) Contrastive_loss: 0.90965 (0.84094) Loss: 6.3057 (6.3227)
|
| 293 |
+
2025-05-07,12:57:42 | WARNING | Handling webdataset error (OSError('image file is truncated (24 bytes not processed)')). Ignoring.
|
| 294 |
+
2025-05-07,13:04:27 | INFO | Train Epoch: 1 [ 35667968/128008192 (28%)] Data (t): 0.383 Batch (t): 5.613, 2930.08/s, 183.130/s/gpu LR: 0.000783 Logit Scale: 75.856 Class_loss: 5.4195 (5.4783) Contrastive_loss: 0.85761 (0.84187) Loss: 6.2771 (6.3202)
|
| 295 |
+
2025-05-07,13:15:43 | WARNING | Handling webdataset error (OSError('image file is truncated (186 bytes not processed)')). Ignoring.
|
| 296 |
+
2025-05-07,13:16:24 | INFO | Train Epoch: 1 [ 37765120/128008192 (30%)] Data (t): 0.376 Batch (t): 5.596, 2938.21/s, 183.638/s/gpu LR: 0.000777 Logit Scale: 75.895 Class_loss: 5.4364 (5.4761) Contrastive_loss: 0.73365 (0.83617) Loss: 6.1701 (6.3123)
|
| 297 |
+
2025-05-07,13:28:25 | INFO | Train Epoch: 1 [ 39862272/128008192 (31%)] Data (t): 0.380 Batch (t): 5.634, 2985.27/s, 186.580/s/gpu LR: 0.000772 Logit Scale: 76.136 Class_loss: 5.4223 (5.4734) Contrastive_loss: 0.80911 (0.83482) Loss: 6.2314 (6.3083)
|
| 298 |
+
2025-05-07,13:40:30 | INFO | Train Epoch: 1 [ 41959424/128008192 (33%)] Data (t): 0.369 Batch (t): 5.667, 2943.40/s, 183.963/s/gpu LR: 0.000767 Logit Scale: 76.346 Class_loss: 5.4294 (5.4713) Contrastive_loss: 0.82847 (0.83452) Loss: 6.2578 (6.3059)
|
| 299 |
+
2025-05-07,13:52:28 | INFO | Train Epoch: 1 [ 44056576/128008192 (34%)] Data (t): 0.371 Batch (t): 5.609, 2984.16/s, 186.510/s/gpu LR: 0.000761 Logit Scale: 76.560 Class_loss: 5.3928 (5.4678) Contrastive_loss: 0.85435 (0.83542) Loss: 6.2472 (6.3032)
|
| 300 |
+
2025-05-07,14:04:24 | INFO | Train Epoch: 1 [ 46153728/128008192 (36%)] Data (t): 0.378 Batch (t): 5.597, 2999.17/s, 187.448/s/gpu LR: 0.000755 Logit Scale: 76.657 Class_loss: 5.4235 (5.4658) Contrastive_loss: 0.89880 (0.83818) Loss: 6.3223 (6.3040)
|
| 301 |
+
2025-05-07,14:16:24 | INFO | Train Epoch: 1 [ 48250880/128008192 (38%)] Data (t): 0.390 Batch (t): 5.619, 3024.85/s, 189.053/s/gpu LR: 0.000750 Logit Scale: 76.963 Class_loss: 5.4334 (5.4645) Contrastive_loss: 0.80629 (0.83685) Loss: 6.2397 (6.3013)
|
| 302 |
+
2025-05-07,14:28:25 | INFO | Train Epoch: 1 [ 50348032/128008192 (39%)] Data (t): 0.385 Batch (t): 5.632, 2934.89/s, 183.430/s/gpu LR: 0.000744 Logit Scale: 77.100 Class_loss: 5.4419 (5.4636) Contrastive_loss: 0.88336 (0.83871) Loss: 6.3252 (6.3023)
|
| 303 |
+
2025-05-07,14:31:17 | WARNING | Handling webdataset error (OSError('image file is truncated (5 bytes not processed)')). Ignoring.
|
| 304 |
+
2025-05-07,14:32:26 | WARNING | Handling webdataset error (OSError('image file is truncated (14 bytes not processed)')). Ignoring.
|
| 305 |
+
2025-05-07,14:40:25 | INFO | Train Epoch: 1 [ 52445184/128008192 (41%)] Data (t): 0.381 Batch (t): 5.624, 3006.25/s, 187.890/s/gpu LR: 0.000738 Logit Scale: 77.219 Class_loss: 5.4234 (5.4620) Contrastive_loss: 0.74680 (0.83517) Loss: 6.1702 (6.2972)
|
| 306 |
+
2025-05-07,14:52:22 | INFO | Train Epoch: 1 [ 54542336/128008192 (43%)] Data (t): 0.380 Batch (t): 5.606, 2930.25/s, 183.141/s/gpu LR: 0.000733 Logit Scale: 77.456 Class_loss: 5.3773 (5.4589) Contrastive_loss: 0.80585 (0.83409) Loss: 6.1831 (6.2930)
|
| 307 |
+
2025-05-07,15:04:23 | INFO | Train Epoch: 1 [ 56639488/128008192 (44%)] Data (t): 0.379 Batch (t): 5.630, 2959.54/s, 184.971/s/gpu LR: 0.000727 Logit Scale: 77.591 Class_loss: 5.4248 (5.4577) Contrastive_loss: 0.79659 (0.83275) Loss: 6.2214 (6.2904)
|
| 308 |
+
2025-05-07,15:08:27 | WARNING | Handling webdataset error (OSError('image file is truncated (76 bytes not processed)')). Ignoring.
|
| 309 |
+
2025-05-07,15:08:47 | WARNING | Handling webdataset error (OSError('image file is truncated (1 bytes not processed)')). Ignoring.
|
| 310 |
+
2025-05-07,15:16:22 | INFO | Train Epoch: 1 [ 58736640/128008192 (46%)] Data (t): 0.370 Batch (t): 5.617, 2981.45/s, 186.341/s/gpu LR: 0.000721 Logit Scale: 77.658 Class_loss: 5.3327 (5.4534) Contrastive_loss: 0.78924 (0.83125) Loss: 6.1219 (6.2846)
|
| 311 |
+
2025-05-07,15:18:35 | WARNING | Handling webdataset error (OSError('image file is truncated (7 bytes not processed)')). Ignoring.
|
| 312 |
+
2025-05-07,15:28:23 | INFO | Train Epoch: 1 [ 60833792/128008192 (48%)] Data (t): 0.383 Batch (t): 5.637, 2915.24/s, 182.203/s/gpu LR: 0.000715 Logit Scale: 77.842 Class_loss: 5.3625 (5.4503) Contrastive_loss: 0.87880 (0.83283) Loss: 6.2413 (6.2832)
|
| 313 |
+
2025-05-07,15:40:24 | INFO | Train Epoch: 1 [ 62930944/128008192 (49%)] Data (t): 0.383 Batch (t): 5.629, 2930.39/s, 183.149/s/gpu LR: 0.000709 Logit Scale: 77.905 Class_loss: 5.3746 (5.4479) Contrastive_loss: 0.92276 (0.83573) Loss: 6.2973 (6.2836)
|
| 314 |
+
2025-05-07,15:49:20 | WARNING | Handling webdataset error (OSError('image file is truncated (5 bytes not processed)')). Ignoring.
|
| 315 |
+
2025-05-07,15:49:37 | WARNING | Handling webdataset error (OSError('image file is truncated (28 bytes not processed)')). Ignoring.
|
| 316 |
+
2025-05-07,15:49:50 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
|
| 317 |
+
2025-05-07,15:52:25 | INFO | Train Epoch: 1 [ 65028096/128008192 (51%)] Data (t): 0.382 Batch (t): 5.633, 2994.84/s, 187.177/s/gpu LR: 0.000703 Logit Scale: 77.901 Class_loss: 5.3819 (5.4458) Contrastive_loss: 0.86842 (0.83675) Loss: 6.2504 (6.2826)
|
| 318 |
+
2025-05-07,15:58:15 | WARNING | Handling webdataset error (OSError('image file is truncated (12 bytes not processed)')). Ignoring.
|
| 319 |
+
2025-05-07,16:03:37 | WARNING | Handling webdataset error (OSError('image file is truncated (28 bytes not processed)')). Ignoring.
|
| 320 |
+
2025-05-07,16:04:29 | INFO | Train Epoch: 1 [ 67125248/128008192 (52%)] Data (t): 0.382 Batch (t): 5.656, 2940.18/s, 183.761/s/gpu LR: 0.000697 Logit Scale: 78.068 Class_loss: 5.3639 (5.4434) Contrastive_loss: 0.89541 (0.83853) Loss: 6.2593 (6.2819)
|
| 321 |
+
2025-05-07,16:16:27 | INFO | Train Epoch: 1 [ 69222400/128008192 (54%)] Data (t): 0.378 Batch (t): 5.611, 2884.54/s, 180.284/s/gpu LR: 0.000691 Logit Scale: 78.200 Class_loss: 5.3251 (5.4399) Contrastive_loss: 0.83057 (0.83830) Loss: 6.1556 (6.2782)
|
| 322 |
+
2025-05-07,16:28:25 | INFO | Train Epoch: 1 [ 71319552/128008192 (56%)] Data (t): 0.380 Batch (t): 5.608, 3038.34/s, 189.896/s/gpu LR: 0.000685 Logit Scale: 78.342 Class_loss: 5.4693 (5.4407) Contrastive_loss: 0.63865 (0.83259) Loss: 6.1079 (6.2733)
|
| 323 |
+
2025-05-07,16:40:26 | INFO | Train Epoch: 1 [ 73416704/128008192 (57%)] Data (t): 0.434 Batch (t): 5.633, 2845.74/s, 177.859/s/gpu LR: 0.000679 Logit Scale: 78.497 Class_loss: 5.3726 (5.4388) Contrastive_loss: 0.86829 (0.83359) Loss: 6.2409 (6.2724)
|
| 324 |
+
2025-05-07,16:47:58 | WARNING | Handling webdataset error (OSError('image file is truncated (151 bytes not processed)')). Ignoring.
|
| 325 |
+
2025-05-07,16:52:27 | INFO | Train Epoch: 1 [ 75513856/128008192 (59%)] Data (t): 0.380 Batch (t): 5.632, 2887.35/s, 180.459/s/gpu LR: 0.000673 Logit Scale: 78.679 Class_loss: 5.3796 (5.4372) Contrastive_loss: 0.88780 (0.83505) Loss: 6.2674 (6.2723)
|
| 326 |
+
2025-05-07,17:04:27 | INFO | Train Epoch: 1 [ 77611008/128008192 (61%)] Data (t): 0.379 Batch (t): 5.627, 2875.65/s, 179.728/s/gpu LR: 0.000667 Logit Scale: 78.777 Class_loss: 5.4398 (5.4373) Contrastive_loss: 0.84522 (0.83532) Loss: 6.2851 (6.2726)
|
| 327 |
+
2025-05-07,17:16:26 | INFO | Train Epoch: 1 [ 79708160/128008192 (62%)] Data (t): 0.378 Batch (t): 5.617, 2947.50/s, 184.219/s/gpu LR: 0.000661 Logit Scale: 78.952 Class_loss: 5.3780 (5.4358) Contrastive_loss: 0.73445 (0.83273) Loss: 6.1124 (6.2685)
|
| 328 |
+
2025-05-07,17:28:25 | INFO | Train Epoch: 1 [ 81805312/128008192 (64%)] Data (t): 0.384 Batch (t): 5.620, 2897.19/s, 181.074/s/gpu LR: 0.000654 Logit Scale: 79.009 Class_loss: 5.3446 (5.4335) Contrastive_loss: 0.78496 (0.83154) Loss: 6.1295 (6.2650)
|
| 329 |
+
2025-05-07,17:40:23 | INFO | Train Epoch: 1 [ 83902464/128008192 (66%)] Data (t): 0.496 Batch (t): 5.607, 2994.08/s, 187.130/s/gpu LR: 0.000648 Logit Scale: 79.182 Class_loss: 5.3789 (5.4322) Contrastive_loss: 0.87432 (0.83258) Loss: 6.2532 (6.2647)
|
| 330 |
+
2025-05-07,17:52:20 | INFO | Train Epoch: 1 [ 85999616/128008192 (67%)] Data (t): 0.383 Batch (t): 5.603, 2922.43/s, 182.652/s/gpu LR: 0.000642 Logit Scale: 79.156 Class_loss: 5.3532 (5.4303) Contrastive_loss: 0.76387 (0.83094) Loss: 6.1171 (6.2612)
|
| 331 |
+
2025-05-07,18:04:17 | INFO | Train Epoch: 1 [ 88096768/128008192 (69%)] Data (t): 0.348 Batch (t): 5.605, 2942.97/s, 183.936/s/gpu LR: 0.000636 Logit Scale: 79.389 Class_loss: 5.3705 (5.4289) Contrastive_loss: 0.70131 (0.82793) Loss: 6.0718 (6.2568)
|
| 332 |
+
2025-05-07,18:16:17 | INFO | Train Epoch: 1 [ 90193920/128008192 (70%)] Data (t): 0.348 Batch (t): 5.621, 2868.75/s, 179.297/s/gpu LR: 0.000629 Logit Scale: 79.495 Class_loss: 5.3877 (5.4280) Contrastive_loss: 0.74144 (0.82596) Loss: 6.1291 (6.2539)
|
| 333 |
+
2025-05-07,18:28:16 | INFO | Train Epoch: 1 [ 92291072/128008192 (72%)] Data (t): 0.380 Batch (t): 5.620, 2886.83/s, 180.427/s/gpu LR: 0.000623 Logit Scale: 79.545 Class_loss: 5.3969 (5.4273) Contrastive_loss: 0.91162 (0.82787) Loss: 6.3085 (6.2551)
|
| 334 |
+
2025-05-07,18:33:35 | WARNING | Handling webdataset error (OSError('image file is truncated (99 bytes not processed)')). Ignoring.
|
| 335 |
+
2025-05-07,18:35:41 | WARNING | Handling webdataset error (OSError('image file is truncated (45 bytes not processed)')). Ignoring.
|
| 336 |
+
2025-05-07,18:37:52 | WARNING | Handling webdataset error (OSError('image file is truncated (55 bytes not processed)')). Ignoring.
|
| 337 |
+
2025-05-07,18:40:12 | INFO | Train Epoch: 1 [ 94388224/128008192 (74%)] Data (t): 0.380 Batch (t): 5.592, 2886.75/s, 180.422/s/gpu LR: 0.000617 Logit Scale: 79.685 Class_loss: 5.4197 (5.4271) Contrastive_loss: 0.71056 (0.82532) Loss: 6.1303 (6.2524)
|
| 338 |
+
2025-05-07,18:52:15 | INFO | Train Epoch: 1 [ 96485376/128008192 (75%)] Data (t): 0.377 Batch (t): 5.649, 2869.22/s, 179.326/s/gpu LR: 0.000610 Logit Scale: 79.818 Class_loss: 5.3510 (5.4255) Contrastive_loss: 0.80795 (0.82495) Loss: 6.1590 (6.2504)
|
| 339 |
+
2025-05-07,18:52:55 | WARNING | Handling webdataset error (OSError('image file is truncated (33 bytes not processed)')). Ignoring.
|
| 340 |
+
2025-05-07,19:04:15 | INFO | Train Epoch: 1 [ 98582528/128008192 (77%)] Data (t): 0.379 Batch (t): 5.620, 2939.49/s, 183.718/s/gpu LR: 0.000604 Logit Scale: 79.887 Class_loss: 5.2500 (5.4218) Contrastive_loss: 0.82735 (0.82500) Loss: 6.0773 (6.2468)
|
| 341 |
+
2025-05-07,19:16:13 | INFO | Train Epoch: 1 [100679680/128008192 (79%)] Data (t): 0.377 Batch (t): 5.616, 2946.90/s, 184.181/s/gpu LR: 0.000597 Logit Scale: 79.949 Class_loss: 5.3328 (5.4200) Contrastive_loss: 0.73090 (0.82308) Loss: 6.0637 (6.2431)
|
| 342 |
+
2025-05-07,19:19:21 | WARNING | Handling webdataset error (OSError('image file is truncated (0 bytes not processed)')). Ignoring.
|
| 343 |
+
2025-05-07,19:23:40 | WARNING | Handling webdataset error (OSError('image file is truncated (108 bytes not processed)')). Ignoring.
|
| 344 |
+
2025-05-07,19:28:11 | INFO | Train Epoch: 1 [102776832/128008192 (80%)] Data (t): 0.377 Batch (t): 5.608, 2973.11/s, 185.819/s/gpu LR: 0.000591 Logit Scale: 80.129 Class_loss: 5.3493 (5.4186) Contrastive_loss: 0.76891 (0.82199) Loss: 6.1182 (6.2406)
|
| 345 |
+
2025-05-07,19:31:08 | WARNING | Handling webdataset error (OSError('image file is truncated (85 bytes not processed)')). Ignoring.
|
| 346 |
+
2025-05-07,19:37:04 | WARNING | Handling webdataset error (OSError('image file is truncated (25 bytes not processed)')). Ignoring.
|
| 347 |
+
2025-05-07,19:40:11 | INFO | Train Epoch: 1 [104873984/128008192 (82%)] Data (t): 0.374 Batch (t): 5.622, 2878.58/s, 179.911/s/gpu LR: 0.000585 Logit Scale: 80.251 Class_loss: 5.3527 (5.4173) Contrastive_loss: 0.77887 (0.82115) Loss: 6.1316 (6.2385)
|
| 348 |
+
2025-05-07,19:52:10 | INFO | Train Epoch: 1 [106971136/128008192 (84%)] Data (t): 0.384 Batch (t): 5.621, 2888.74/s, 180.546/s/gpu LR: 0.000578 Logit Scale: 80.335 Class_loss: 5.3088 (5.4152) Contrastive_loss: 0.84990 (0.82170) Loss: 6.1587 (6.2369)
|
| 349 |
+
2025-05-07,20:01:25 | WARNING | Handling webdataset error (OSError('image file is truncated (67 bytes not processed)')). Ignoring.
|
| 350 |
+
2025-05-07,20:04:09 | INFO | Train Epoch: 1 [109068288/128008192 (85%)] Data (t): 0.383 Batch (t): 5.618, 2946.38/s, 184.148/s/gpu LR: 0.000572 Logit Scale: 80.492 Class_loss: 5.2615 (5.4123) Contrastive_loss: 0.87412 (0.82269) Loss: 6.1357 (6.2350)
|
| 351 |
+
2025-05-07,20:16:09 | INFO | Train Epoch: 1 [111165440/128008192 (87%)] Data (t): 0.380 Batch (t): 5.624, 2925.22/s, 182.826/s/gpu LR: 0.000565 Logit Scale: 80.589 Class_loss: 5.3806 (5.4117) Contrastive_loss: 0.68782 (0.82019) Loss: 6.0684 (6.2319)
|
| 352 |
+
2025-05-07,20:21:24 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
|
| 353 |
+
2025-05-07,20:24:45 | WARNING | Handling webdataset error (OSError('image file is truncated (21 bytes not processed)')). Ignoring.
|
| 354 |
+
2025-05-07,20:28:10 | INFO | Train Epoch: 1 [113262592/128008192 (88%)] Data (t): 0.449 Batch (t): 5.633, 2808.76/s, 175.547/s/gpu LR: 0.000559 Logit Scale: 80.739 Class_loss: 5.3211 (5.4101) Contrastive_loss: 0.60587 (0.81630) Loss: 5.9269 (6.2264)
|
| 355 |
+
2025-05-07,20:40:08 | INFO | Train Epoch: 1 [115359744/128008192 (90%)] Data (t): 0.431 Batch (t): 5.609, 2699.57/s, 168.723/s/gpu LR: 0.000552 Logit Scale: 80.997 Class_loss: 5.2988 (5.4081) Contrastive_loss: 0.62347 (0.81285) Loss: 5.9223 (6.2209)
|
| 356 |
+
2025-05-07,20:49:21 | WARNING | Handling webdataset error (OSError('image file is truncated (88 bytes not processed)')). Ignoring.
|
| 357 |
+
2025-05-07,20:52:13 | INFO | Train Epoch: 1 [117456896/128008192 (92%)] Data (t): 0.361 Batch (t): 5.661, 2929.30/s, 183.081/s/gpu LR: 0.000546 Logit Scale: 81.086 Class_loss: 5.2301 (5.4050) Contrastive_loss: 0.83269 (0.81320) Loss: 6.0628 (6.2182)
|
| 358 |
+
2025-05-07,21:02:45 | WARNING | Handling webdataset error (OSError('image file is truncated (38 bytes not processed)')). Ignoring.
|
| 359 |
+
2025-05-07,21:04:11 | INFO | Train Epoch: 1 [119554048/128008192 (93%)] Data (t): 0.378 Batch (t): 5.616, 2854.81/s, 178.426/s/gpu LR: 0.000539 Logit Scale: 81.118 Class_loss: 5.2659 (5.4026) Contrastive_loss: 0.82355 (0.81338) Loss: 6.0894 (6.2160)
|
| 360 |
+
2025-05-07,21:13:53 | WARNING | Handling webdataset error (OSError('image file is truncated (31 bytes not processed)')). Ignoring.
|
| 361 |
+
2025-05-07,21:16:14 | INFO | Train Epoch: 1 [121651200/128008192 (95%)] Data (t): 0.386 Batch (t): 5.644, 2863.34/s, 178.958/s/gpu LR: 0.000533 Logit Scale: 81.239 Class_loss: 5.3024 (5.4009) Contrastive_loss: 0.71169 (0.81166) Loss: 6.0141 (6.2125)
|
| 362 |
+
2025-05-07,21:21:24 | WARNING | Handling webdataset error (OSError('image file is truncated (26 bytes not processed)')). Ignoring.
|
| 363 |
+
2025-05-07,21:22:07 | WARNING | Handling webdataset error (OSError('image file is truncated (89 bytes not processed)')). Ignoring.
|
| 364 |
+
2025-05-07,21:28:13 | INFO | Train Epoch: 1 [123748352/128008192 (97%)] Data (t): 0.386 Batch (t): 5.619, 2951.10/s, 184.444/s/gpu LR: 0.000526 Logit Scale: 81.417 Class_loss: 5.3248 (5.3996) Contrastive_loss: 0.74739 (0.81058) Loss: 6.0722 (6.2102)
|
| 365 |
+
2025-05-07,21:40:16 | INFO | Train Epoch: 1 [125845504/128008192 (98%)] Data (t): 0.379 Batch (t): 5.646, 2897.09/s, 181.068/s/gpu LR: 0.000520 Logit Scale: 81.438 Class_loss: 5.3063 (5.3981) Contrastive_loss: 0.70879 (0.80892) Loss: 6.0151 (6.2070)
|
| 366 |
+
2025-05-07,21:48:14 | WARNING | Handling webdataset error (OSError('image file is truncated (1 bytes not processed)')). Ignoring.
|
| 367 |
+
2025-05-07,21:52:20 | INFO | Train Epoch: 1 [127942656/128008192 (100%)] Data (t): 0.381 Batch (t): 5.655, 2898.02/s, 181.126/s/gpu LR: 0.000513 Logit Scale: 81.568 Class_loss: 5.2757 (5.3961) Contrastive_loss: 0.70257 (0.80720) Loss: 5.9782 (6.2033)
|
| 368 |
+
2025-05-07,21:52:42 | INFO | Train Epoch: 1 [128008192/128008192 (100%)] Data (t): 0.373 Batch (t): 5.529, 2978.09/s, 186.131/s/gpu LR: 0.000513 Logit Scale: 81.557 Class_loss: 5.3150 (5.3948) Contrastive_loss: 0.67616 (0.80512) Loss: 5.9912 (6.1999)
|
| 369 |
+
2025-05-07,21:52:50 | INFO | Start epoch 2
|
| 370 |
+
2025-05-07,21:53:02 | INFO | Train Epoch: 2 [ 16384/128008192 (0%)] Data (t): 7.371 Batch (t): 11.798, 1388.69/s, 86.7930/s/gpu LR: 0.000513 Logit Scale: 81.549 Class_loss: 5.2344 (5.2344) Contrastive_loss: 0.79251 (0.79251) Loss: 6.0269 (6.0269)
|
| 371 |
+
2025-05-07,21:57:05 | WARNING | Handling webdataset error (OSError('image file is truncated (9 bytes not processed)')). Ignoring.
|
| 372 |
+
2025-05-07,22:04:58 | INFO | Train Epoch: 2 [ 2113536/128008192 (2%)] Data (t): 0.509 Batch (t): 5.594, 2979.46/s, 186.216/s/gpu LR: 0.000506 Logit Scale: 81.737 Class_loss: 5.3498 (5.2921) Contrastive_loss: 0.80754 (0.80002) Loss: 6.1573 (6.0921)
|
| 373 |
+
2025-05-07,22:11:49 | WARNING | Handling webdataset error (OSError('image file is truncated (101 bytes not processed)')). Ignoring.
|
| 374 |
+
2025-05-07,22:16:54 | INFO | Train Epoch: 2 [ 4210688/128008192 (3%)] Data (t): 0.376 Batch (t): 5.596, 2941.02/s, 183.813/s/gpu LR: 0.000500 Logit Scale: 81.925 Class_loss: 5.3262 (5.3035) Contrastive_loss: 0.52220 (0.70742) Loss: 5.8484 (6.0109)
|
| 375 |
+
2025-05-07,22:27:11 | WARNING | Handling webdataset error (OSError('image file is truncated (46 bytes not processed)')). Ignoring.
|
| 376 |
+
2025-05-07,22:28:51 | INFO | Train Epoch: 2 [ 6307840/128008192 (5%)] Data (t): 0.375 Batch (t): 5.599, 2969.22/s, 185.576/s/gpu LR: 0.000493 Logit Scale: 82.110 Class_loss: 5.2499 (5.2901) Contrastive_loss: 0.80884 (0.73277) Loss: 6.0587 (6.0228)
|
| 377 |
+
2025-05-07,22:34:19 | WARNING | Handling webdataset error (OSError('image file is truncated (3 bytes not processed)')). Ignoring.
|
| 378 |
+
2025-05-07,22:35:55 | WARNING | Handling webdataset error (OSError('image file is truncated (9 bytes not processed)')). Ignoring.
|
| 379 |
+
2025-05-07,22:39:44 | WARNING | Handling webdataset error (OSError('image file is truncated (61 bytes not processed)')). Ignoring.
|
| 380 |
+
2025-05-07,22:40:53 | INFO | Train Epoch: 2 [ 8404992/128008192 (7%)] Data (t): 0.384 Batch (t): 5.639, 2930.46/s, 183.154/s/gpu LR: 0.000487 Logit Scale: 82.285 Class_loss: 5.2739 (5.2868) Contrastive_loss: 0.72890 (0.73200) Loss: 6.0028 (6.0188)
|
| 381 |
+
2025-05-07,22:52:52 | INFO | Train Epoch: 2 [ 10502144/128008192 (8%)] Data (t): 0.389 Batch (t): 5.619, 2933.31/s, 183.332/s/gpu LR: 0.000480 Logit Scale: 82.429 Class_loss: 5.2448 (5.2798) Contrastive_loss: 0.82383 (0.74730) Loss: 6.0686 (6.0271)
|
| 382 |
+
2025-05-07,23:04:50 | INFO | Train Epoch: 2 [ 12599296/128008192 (10%)] Data (t): 0.380 Batch (t): 5.613, 2916.75/s, 182.297/s/gpu LR: 0.000474 Logit Scale: 82.509 Class_loss: 5.2763 (5.2793) Contrastive_loss: 0.67358 (0.73677) Loss: 5.9498 (6.0161)
|
| 383 |
+
2025-05-07,23:16:46 | INFO | Train Epoch: 2 [ 14696448/128008192 (11%)] Data (t): 0.671 Batch (t): 5.594, 2901.75/s, 181.359/s/gpu LR: 0.000467 Logit Scale: 82.720 Class_loss: 5.2898 (5.2806) Contrastive_loss: 0.67115 (0.72857) Loss: 5.9609 (6.0092)
|
| 384 |
+
2025-05-07,23:28:44 | INFO | Train Epoch: 2 [ 16793600/128008192 (13%)] Data (t): 0.372 Batch (t): 5.611, 2890.20/s, 180.637/s/gpu LR: 0.000461 Logit Scale: 82.792 Class_loss: 5.2973 (5.2825) Contrastive_loss: 0.85458 (0.74257) Loss: 6.1519 (6.0250)
|
| 385 |
+
2025-05-07,23:32:43 | WARNING | Handling webdataset error (OSError('broken data stream when reading image file')). Ignoring.
|
| 386 |
+
2025-05-07,23:36:27 | WARNING | Handling webdataset error (OSError('image file is truncated (7 bytes not processed)')). Ignoring.
|
| 387 |
+
2025-05-07,23:40:47 | INFO | Train Epoch: 2 [ 18890752/128008192 (15%)] Data (t): 0.373 Batch (t): 5.641, 2900.12/s, 181.257/s/gpu LR: 0.000454 Logit Scale: 82.884 Class_loss: 5.2789 (5.2821) Contrastive_loss: 0.67827 (0.73614) Loss: 5.9571 (6.0182)
|
| 388 |
+
2025-05-07,23:43:18 | WARNING | Handling webdataset error (OSError('image file is truncated (88 bytes not processed)')). Ignoring.
|
| 389 |
+
2025-05-07,23:52:48 | INFO | Train Epoch: 2 [ 20987904/128008192 (16%)] Data (t): 0.379 Batch (t): 5.636, 2922.07/s, 182.629/s/gpu LR: 0.000447 Logit Scale: 83.058 Class_loss: 5.2595 (5.2801) Contrastive_loss: 0.67788 (0.73084) Loss: 5.9373 (6.0109)
|
| 390 |
+
2025-05-07,23:58:24 | WARNING | Handling webdataset error (OSError('image file is truncated (47 bytes not processed)')). Ignoring.
|
| 391 |
+
2025-05-08,00:04:48 | INFO | Train Epoch: 2 [ 23085056/128008192 (18%)] Data (t): 0.375 Batch (t): 5.629, 2994.04/s, 187.127/s/gpu LR: 0.000441 Logit Scale: 83.156 Class_loss: 5.2882 (5.2807) Contrastive_loss: 0.65210 (0.72428) Loss: 5.9403 (6.0050)
|
| 392 |
+
2025-05-08,00:07:16 | WARNING | Handling webdataset error (OSError('image file is truncated (82 bytes not processed)')). Ignoring.
|
| 393 |
+
2025-05-08,00:16:50 | INFO | Train Epoch: 2 [ 25182208/128008192 (20%)] Data (t): 0.374 Batch (t): 5.637, 2971.32/s, 185.708/s/gpu LR: 0.000435 Logit Scale: 83.317 Class_loss: 5.2205 (5.2761) Contrastive_loss: 0.87624 (0.73597) Loss: 6.0967 (6.0121)
|
| 394 |
+
2025-05-08,00:28:51 | INFO | Train Epoch: 2 [ 27279360/128008192 (21%)] Data (t): 0.381 Batch (t): 5.629, 2934.05/s, 183.378/s/gpu LR: 0.000428 Logit Scale: 83.493 Class_loss: 5.2766 (5.2761) Contrastive_loss: 0.63712 (0.72891) Loss: 5.9137 (6.0050)
|
| 395 |
+
2025-05-08,00:40:49 | INFO | Train Epoch: 2 [ 29376512/128008192 (23%)] Data (t): 0.377 Batch (t): 5.610, 2916.98/s, 182.311/s/gpu LR: 0.000422 Logit Scale: 83.557 Class_loss: 5.3062 (5.2781) Contrastive_loss: 0.68775 (0.72616) Loss: 5.9940 (6.0043)
|
| 396 |
+
2025-05-08,00:52:45 | INFO | Train Epoch: 2 [ 31473664/128008192 (25%)] Data (t): 0.373 Batch (t): 5.597, 2931.01/s, 183.188/s/gpu LR: 0.000415 Logit Scale: 83.730 Class_loss: 5.2464 (5.2762) Contrastive_loss: 0.79413 (0.73041) Loss: 6.0405 (6.0066)
|
| 397 |
+
2025-05-08,01:04:43 | INFO | Train Epoch: 2 [ 33570816/128008192 (26%)] Data (t): 0.378 Batch (t): 5.608, 2875.14/s, 179.696/s/gpu LR: 0.000409 Logit Scale: 83.844 Class_loss: 5.2075 (5.2721) Contrastive_loss: 0.80586 (0.73485) Loss: 6.0133 (6.0070)
|
| 398 |
+
2025-05-08,01:16:40 | INFO | Train Epoch: 2 [ 35667968/128008192 (28%)] Data (t): 0.378 Batch (t): 5.603, 2972.28/s, 185.767/s/gpu LR: 0.000402 Logit Scale: 83.980 Class_loss: 5.2002 (5.2681) Contrastive_loss: 0.73259 (0.73472) Loss: 5.9328 (6.0028)
|
| 399 |
+
2025-05-08,01:28:37 | INFO | Train Epoch: 2 [ 37765120/128008192 (30%)] Data (t): 0.374 Batch (t): 5.602, 2959.92/s, 184.995/s/gpu LR: 0.000396 Logit Scale: 84.127 Class_loss: 5.2484 (5.2671) Contrastive_loss: 0.63592 (0.72952) Loss: 5.8844 (5.9966)
|
| 400 |
+
2025-05-08,01:40:37 | INFO | Train Epoch: 2 [ 39862272/128008192 (31%)] Data (t): 0.379 Batch (t): 5.621, 2919.92/s, 182.495/s/gpu LR: 0.000389 Logit Scale: 84.263 Class_loss: 5.3205 (5.2698) Contrastive_loss: 0.64314 (0.72521) Loss: 5.9637 (5.9950)
|
| 401 |
+
2025-05-08,01:41:53 | WARNING | Handling webdataset error (OSError('image file is truncated (28 bytes not processed)')). Ignoring.
|
| 402 |
+
2025-05-08,01:45:16 | WARNING | Handling webdataset error (OSError('image file is truncated (76 bytes not processed)')). Ignoring.
|
| 403 |
+
2025-05-08,01:46:16 | WARNING | Handling webdataset error (OSError('image file is truncated (230 bytes not processed)')). Ignoring.
|
| 404 |
+
2025-05-08,01:52:39 | INFO | Train Epoch: 2 [ 41959424/128008192 (33%)] Data (t): 0.381 Batch (t): 5.644, 2965.43/s, 185.339/s/gpu LR: 0.000383 Logit Scale: 84.434 Class_loss: 5.1896 (5.2659) Contrastive_loss: 0.70490 (0.72424) Loss: 5.8945 (5.9902)
|
| 405 |
+
2025-05-08,01:55:43 | WARNING | Handling webdataset error (OSError('image file is truncated (12 bytes not processed)')). Ignoring.
|
| 406 |
+
2025-05-08,02:04:47 | INFO | Train Epoch: 2 [ 44056576/128008192 (34%)] Data (t): 0.415 Batch (t): 5.684, 3415.69/s, 213.480/s/gpu LR: 0.000377 Logit Scale: 84.640 Class_loss: 5.2608 (5.2657) Contrastive_loss: 0.59660 (0.71844) Loss: 5.8574 (5.9841)
|
| 407 |
+
2025-05-08,02:16:45 | INFO | Train Epoch: 2 [ 46153728/128008192 (36%)] Data (t): 0.498 Batch (t): 5.612, 2892.59/s, 180.787/s/gpu LR: 0.000370 Logit Scale: 84.764 Class_loss: 5.1890 (5.2624) Contrastive_loss: 0.85909 (0.72455) Loss: 6.0481 (5.9869)
|
| 408 |
+
2025-05-08,02:18:01 | WARNING | Handling webdataset error (OSError('image file is truncated (19 bytes not processed)')). Ignoring.
|
| 409 |
+
2025-05-08,02:20:51 | WARNING | Handling webdataset error (OSError('image file is truncated (7 bytes not processed)')). Ignoring.
|
| 410 |
+
2025-05-08,02:28:44 | INFO | Train Epoch: 2 [ 48250880/128008192 (38%)] Data (t): 0.384 Batch (t): 5.617, 2919.83/s, 182.489/s/gpu LR: 0.000364 Logit Scale: 84.915 Class_loss: 5.2465 (5.2617) Contrastive_loss: 0.81985 (0.72852) Loss: 6.0663 (5.9902)
|
| 411 |
+
2025-05-08,02:40:41 | INFO | Train Epoch: 2 [ 50348032/128008192 (39%)] Data (t): 0.391 Batch (t): 5.606, 2919.13/s, 182.446/s/gpu LR: 0.000358 Logit Scale: 84.943 Class_loss: 5.2682 (5.2620) Contrastive_loss: 0.64299 (0.72510) Loss: 5.9112 (5.9871)
|
| 412 |
+
2025-05-08,02:42:59 | WARNING | Handling webdataset error (OSError('image file is truncated (3 bytes not processed)')). Ignoring.
|
| 413 |
+
2025-05-08,02:43:52 | WARNING | Handling webdataset error (OSError('image file is truncated (43 bytes not processed)')). Ignoring.
|
| 414 |
+
2025-05-08,02:52:40 | INFO | Train Epoch: 2 [ 52445184/128008192 (41%)] Data (t): 0.377 Batch (t): 5.611, 2966.18/s, 185.386/s/gpu LR: 0.000352 Logit Scale: 85.199 Class_loss: 5.2174 (5.2603) Contrastive_loss: 0.63427 (0.72161) Loss: 5.8517 (5.9819)
|
| 415 |
+
2025-05-08,03:04:40 | INFO | Train Epoch: 2 [ 54542336/128008192 (43%)] Data (t): 0.376 Batch (t): 5.625, 2915.26/s, 182.204/s/gpu LR: 0.000345 Logit Scale: 85.351 Class_loss: 5.2020 (5.2581) Contrastive_loss: 0.65317 (0.71907) Loss: 5.8551 (5.9772)
|
| 416 |
+
2025-05-08,03:16:38 | INFO | Train Epoch: 2 [ 56639488/128008192 (44%)] Data (t): 0.379 Batch (t): 5.610, 2876.07/s, 179.754/s/gpu LR: 0.000339 Logit Scale: 85.468 Class_loss: 5.2535 (5.2579) Contrastive_loss: 0.61386 (0.71532) Loss: 5.8674 (5.9732)
|
| 417 |
+
2025-05-08,03:28:37 | INFO | Train Epoch: 2 [ 58736640/128008192 (46%)] Data (t): 0.381 Batch (t): 5.620, 2959.93/s, 184.996/s/gpu LR: 0.000333 Logit Scale: 85.585 Class_loss: 5.2043 (5.2561) Contrastive_loss: 0.67879 (0.71406) Loss: 5.8831 (5.9701)
|
| 418 |
+
2025-05-08,03:40:36 | INFO | Train Epoch: 2 [ 60833792/128008192 (48%)] Data (t): 0.381 Batch (t): 5.618, 3005.90/s, 187.869/s/gpu LR: 0.000327 Logit Scale: 85.708 Class_loss: 5.2884 (5.2572) Contrastive_loss: 0.62679 (0.71115) Loss: 5.9152 (5.9683)
|
| 419 |
+
2025-05-08,03:45:17 | WARNING | Handling webdataset error (OSError('image file is truncated (59 bytes not processed)')). Ignoring.
|
| 420 |
+
2025-05-08,03:52:37 | INFO | Train Epoch: 2 [ 62930944/128008192 (49%)] Data (t): 0.381 Batch (t): 5.632, 2735.45/s, 170.966/s/gpu LR: 0.000321 Logit Scale: 85.827 Class_loss: 5.2703 (5.2576) Contrastive_loss: 0.83742 (0.71522) Loss: 6.1077 (5.9728)
|
| 421 |
+
2025-05-08,03:56:43 | WARNING | Handling webdataset error (OSError('image file is truncated (8 bytes not processed)')). Ignoring.
|
| 422 |
+
2025-05-08,04:04:38 | INFO | Train Epoch: 2 [ 65028096/128008192 (51%)] Data (t): 0.380 Batch (t): 5.634, 2908.63/s, 181.789/s/gpu LR: 0.000315 Logit Scale: 86.026 Class_loss: 5.2236 (5.2565) Contrastive_loss: 0.67039 (0.71382) Loss: 5.8940 (5.9703)
|
| 423 |
+
2025-05-08,04:08:29 | WARNING | Handling webdataset error (OSError('image file is truncated (54 bytes not processed)')). Ignoring.
|
| 424 |
+
2025-05-08,04:11:35 | WARNING | Handling webdataset error (OSError('image file is truncated (22 bytes not processed)')). Ignoring.
|
| 425 |
+
2025-05-08,04:16:40 | INFO | Train Epoch: 2 [ 67125248/128008192 (52%)] Data (t): 0.387 Batch (t): 5.641, 2857.20/s, 178.575/s/gpu LR: 0.000309 Logit Scale: 86.076 Class_loss: 5.1708 (5.2539) Contrastive_loss: 0.62589 (0.71116) Loss: 5.7967 (5.9651)
|
| 426 |
+
2025-05-08,04:28:35 | INFO | Train Epoch: 2 [ 69222400/128008192 (54%)] Data (t): 0.371 Batch (t): 5.584, 2908.20/s, 181.763/s/gpu LR: 0.000303 Logit Scale: 86.258 Class_loss: 5.1850 (5.2519) Contrastive_loss: 0.59906 (0.70786) Loss: 5.7841 (5.9598)
|
| 427 |
+
2025-05-08,04:40:40 | INFO | Train Epoch: 2 [ 71319552/128008192 (56%)] Data (t): 0.375 Batch (t): 5.663, 2864.86/s, 179.054/s/gpu LR: 0.000297 Logit Scale: 86.376 Class_loss: 5.1988 (5.2504) Contrastive_loss: 0.76567 (0.70951) Loss: 5.9645 (5.9599)
|
| 428 |
+
2025-05-08,04:52:34 | INFO | Train Epoch: 2 [ 73416704/128008192 (57%)] Data (t): 0.357 Batch (t): 5.577, 2900.97/s, 181.310/s/gpu LR: 0.000291 Logit Scale: 86.542 Class_loss: 5.1475 (5.2475) Contrastive_loss: 0.69326 (0.70906) Loss: 5.8407 (5.9566)
|
| 429 |
+
2025-05-08,04:58:20 | WARNING | Handling webdataset error (OSError('image file is truncated (34 bytes not processed)')). Ignoring.
|
| 430 |
+
2025-05-08,05:04:33 | INFO | Train Epoch: 2 [ 75513856/128008192 (59%)] Data (t): 0.381 Batch (t): 5.622, 2945.00/s, 184.063/s/gpu LR: 0.000285 Logit Scale: 86.641 Class_loss: 5.1508 (5.2449) Contrastive_loss: 0.69695 (0.70873) Loss: 5.8478 (5.9536)
|
| 431 |
+
2025-05-08,05:10:37 | WARNING | Handling webdataset error (OSError('image file is truncated (86 bytes not processed)')). Ignoring.
|
| 432 |
+
2025-05-08,05:15:03 | WARNING | Handling webdataset error (OSError('image file is truncated (37 bytes not processed)')). Ignoring.
|
| 433 |
+
2025-05-08,05:16:33 | INFO | Train Epoch: 2 [ 77611008/128008192 (61%)] Data (t): 0.356 Batch (t): 5.622, 2866.58/s, 179.161/s/gpu LR: 0.000279 Logit Scale: 86.807 Class_loss: 5.2481 (5.2450) Contrastive_loss: 0.53480 (0.70415) Loss: 5.7829 (5.9491)
|
| 434 |
+
2025-05-08,05:21:01 | WARNING | Handling webdataset error (OSError('image file is truncated (152 bytes not processed)')). Ignoring.
|
| 435 |
+
2025-05-08,05:23:29 | WARNING | Handling webdataset error (OSError('image file is truncated (80 bytes not processed)')). Ignoring.
|
| 436 |
+
2025-05-08,05:23:43 | WARNING | Handling webdataset error (OSError('image file is truncated (16 bytes not processed)')). Ignoring.
|
| 437 |
+
2025-05-08,05:28:31 | INFO | Train Epoch: 2 [ 79708160/128008192 (62%)] Data (t): 0.381 Batch (t): 5.612, 2902.18/s, 181.387/s/gpu LR: 0.000273 Logit Scale: 86.944 Class_loss: 5.1619 (5.2429) Contrastive_loss: 0.69565 (0.70394) Loss: 5.8575 (5.9468)
|
| 438 |
+
2025-05-08,05:40:31 | INFO | Train Epoch: 2 [ 81805312/128008192 (64%)] Data (t): 0.386 Batch (t): 5.620, 2914.98/s, 182.186/s/gpu LR: 0.000267 Logit Scale: 87.070 Class_loss: 5.1181 (5.2397) Contrastive_loss: 0.79149 (0.70612) Loss: 5.9096 (5.9459)
|
| 439 |
+
2025-05-08,05:42:26 | WARNING | Handling webdataset error (OSError('image file is truncated (60 bytes not processed)')). Ignoring.
|
| 440 |
+
2025-05-08,05:52:31 | INFO | Train Epoch: 2 [ 83902464/128008192 (66%)] Data (t): 0.370 Batch (t): 5.625, 2888.82/s, 180.552/s/gpu LR: 0.000261 Logit Scale: 87.179 Class_loss: 5.2364 (5.2397) Contrastive_loss: 0.73369 (0.70680) Loss: 5.9701 (5.9465)
|
| 441 |
+
2025-05-08,06:04:29 | INFO | Train Epoch: 2 [ 85999616/128008192 (67%)] Data (t): 0.378 Batch (t): 5.609, 2885.78/s, 180.362/s/gpu LR: 0.000256 Logit Scale: 87.374 Class_loss: 5.1710 (5.2380) Contrastive_loss: 0.75078 (0.70784) Loss: 5.9218 (5.9459)
|
| 442 |
+
2025-05-08,06:11:19 | WARNING | Handling webdataset error (OSError('broken data stream when reading image file')). Ignoring.
|
| 443 |
+
2025-05-08,06:16:29 | INFO | Train Epoch: 2 [ 88096768/128008192 (69%)] Data (t): 0.369 Batch (t): 5.625, 3010.77/s, 188.173/s/gpu LR: 0.000250 Logit Scale: 87.582 Class_loss: 5.1887 (5.2369) Contrastive_loss: 0.69575 (0.70756) Loss: 5.8845 (5.9444)
|
| 444 |
+
2025-05-08,06:28:29 | INFO | Train Epoch: 2 [ 90193920/128008192 (70%)] Data (t): 0.369 Batch (t): 5.628, 2756.77/s, 172.298/s/gpu LR: 0.000244 Logit Scale: 87.790 Class_loss: 5.1113 (5.2340) Contrastive_loss: 0.65936 (0.70647) Loss: 5.7706 (5.9405)
|
| 445 |
+
2025-05-08,06:32:13 | WARNING | Handling webdataset error (OSError('image file is truncated (33 bytes not processed)')). Ignoring.
|
| 446 |
+
2025-05-08,06:33:32 | WARNING | Handling webdataset error (OSError('image file is truncated (16 bytes not processed)')). Ignoring.
|
| 447 |
+
2025-05-08,06:40:28 | INFO | Train Epoch: 2 [ 92291072/128008192 (72%)] Data (t): 0.368 Batch (t): 5.620, 2970.53/s, 185.658/s/gpu LR: 0.000239 Logit Scale: 87.928 Class_loss: 5.1776 (5.2328) Contrastive_loss: 0.67725 (0.70582) Loss: 5.8548 (5.9386)
|
| 448 |
+
2025-05-08,06:52:28 | INFO | Train Epoch: 2 [ 94388224/128008192 (74%)] Data (t): 0.390 Batch (t): 5.625, 2902.76/s, 181.422/s/gpu LR: 0.000233 Logit Scale: 88.139 Class_loss: 5.1552 (5.2311) Contrastive_loss: 0.62844 (0.70414) Loss: 5.7837 (5.9352)
|
| 449 |
+
2025-05-08,07:04:28 | INFO | Train Epoch: 2 [ 96485376/128008192 (75%)] Data (t): 0.382 Batch (t): 5.624, 2945.03/s, 184.064/s/gpu LR: 0.000228 Logit Scale: 88.387 Class_loss: 5.1512 (5.2294) Contrastive_loss: 0.74295 (0.70496) Loss: 5.8941 (5.9343)
|
| 450 |
+
2025-05-08,07:15:07 | WARNING | Handling webdataset error (OSError('image file is truncated (26 bytes not processed)')). Ignoring.
|
| 451 |
+
2025-05-08,07:16:29 | INFO | Train Epoch: 2 [ 98582528/128008192 (77%)] Data (t): 0.382 Batch (t): 5.630, 2911.61/s, 181.976/s/gpu LR: 0.000222 Logit Scale: 88.481 Class_loss: 5.2019 (5.2288) Contrastive_loss: 0.65733 (0.70397) Loss: 5.8592 (5.9328)
|
| 452 |
+
2025-05-08,07:28:29 | INFO | Train Epoch: 2 [100679680/128008192 (79%)] Data (t): 0.374 Batch (t): 5.629, 2923.81/s, 182.738/s/gpu LR: 0.000217 Logit Scale: 88.644 Class_loss: 5.1898 (5.2280) Contrastive_loss: 0.79600 (0.70585) Loss: 5.9858 (5.9339)
|
| 453 |
+
2025-05-08,07:40:28 | INFO | Train Epoch: 2 [102776832/128008192 (80%)] Data (t): 0.376 Batch (t): 5.617, 2919.36/s, 182.460/s/gpu LR: 0.000211 Logit Scale: 88.725 Class_loss: 5.0864 (5.2252) Contrastive_loss: 0.73069 (0.70634) Loss: 5.8171 (5.9315)
|
| 454 |
+
2025-05-08,07:52:27 | INFO | Train Epoch: 2 [104873984/128008192 (82%)] Data (t): 0.375 Batch (t): 5.616, 2869.19/s, 179.324/s/gpu LR: 0.000206 Logit Scale: 88.973 Class_loss: 5.1891 (5.2245) Contrastive_loss: 0.69331 (0.70609) Loss: 5.8825 (5.9306)
|
| 455 |
+
2025-05-08,08:04:28 | INFO | Train Epoch: 2 [106971136/128008192 (84%)] Data (t): 0.376 Batch (t): 5.631, 2968.85/s, 185.553/s/gpu LR: 0.000201 Logit Scale: 89.123 Class_loss: 5.1296 (5.2227) Contrastive_loss: 0.73598 (0.70666) Loss: 5.8656 (5.9293)
|
| 456 |
+
2025-05-08,08:07:06 | WARNING | Handling webdataset error (OSError('image file is truncated (66 bytes not processed)')). Ignoring.
|
| 457 |
+
2025-05-08,08:16:29 | INFO | Train Epoch: 2 [109068288/128008192 (85%)] Data (t): 0.385 Batch (t): 5.635, 2937.24/s, 183.577/s/gpu LR: 0.000196 Logit Scale: 89.285 Class_loss: 5.1404 (5.2211) Contrastive_loss: 0.65773 (0.70574) Loss: 5.7982 (5.9268)
|
| 458 |
+
2025-05-08,08:19:33 | WARNING | Handling webdataset error (OSError('image file is truncated (86 bytes not processed)')). Ignoring.
|
| 459 |
+
2025-05-08,08:22:41 | WARNING | Handling webdataset error (OSError('image file is truncated (29 bytes not processed)')). Ignoring.
|
| 460 |
+
2025-05-08,08:26:04 | WARNING | Handling webdataset error (OSError('image file is truncated (80 bytes not processed)')). Ignoring.
|
| 461 |
+
2025-05-08,08:26:18 | WARNING | Handling webdataset error (OSError('image file is truncated (4 bytes not processed)')). Ignoring.
|
| 462 |
+
2025-05-08,08:28:32 | INFO | Train Epoch: 2 [111165440/128008192 (87%)] Data (t): 0.381 Batch (t): 5.646, 2923.81/s, 182.738/s/gpu LR: 0.000190 Logit Scale: 89.396 Class_loss: 5.1308 (5.2194) Contrastive_loss: 0.73641 (0.70631) Loss: 5.8672 (5.9257)
|
| 463 |
+
2025-05-08,08:35:36 | WARNING | Handling webdataset error (OSError('image file is truncated (67 bytes not processed)')). Ignoring.
|
| 464 |
+
2025-05-08,08:36:43 | WARNING | Handling webdataset error (OSError('image file is truncated (19 bytes not processed)')). Ignoring.
|
| 465 |
+
2025-05-08,08:38:22 | WARNING | Handling webdataset error (OSError('image file is truncated (8 bytes not processed)')). Ignoring.
|
| 466 |
+
2025-05-08,08:40:33 | INFO | Train Epoch: 2 [113262592/128008192 (88%)] Data (t): 0.380 Batch (t): 5.629, 2871.35/s, 179.459/s/gpu LR: 0.000185 Logit Scale: 89.563 Class_loss: 5.1812 (5.2187) Contrastive_loss: 0.69934 (0.70618) Loss: 5.8805 (5.9249)
|
| 467 |
+
2025-05-08,08:42:50 | WARNING | Handling webdataset error (OSError('image file is truncated (93 bytes not processed)')). Ignoring.
|
| 468 |
+
2025-05-08,08:52:38 | INFO | Train Epoch: 2 [115359744/128008192 (90%)] Data (t): 0.376 Batch (t): 5.668, 2755.57/s, 172.223/s/gpu LR: 0.000180 Logit Scale: 89.752 Class_loss: 5.1736 (5.2179) Contrastive_loss: 0.58980 (0.70410) Loss: 5.7634 (5.9220)
|
| 469 |
+
2025-05-08,08:58:22 | WARNING | Handling webdataset error (OSError('image file is truncated (8 bytes not processed)')). Ignoring.
|
| 470 |
+
2025-05-08,09:04:37 | INFO | Train Epoch: 2 [117456896/128008192 (92%)] Data (t): 0.376 Batch (t): 5.619, 2918.65/s, 182.416/s/gpu LR: 0.000175 Logit Scale: 89.942 Class_loss: 5.1575 (5.2169) Contrastive_loss: 0.56595 (0.70168) Loss: 5.7234 (5.9185)
|
| 471 |
+
2025-05-08,09:16:36 | INFO | Train Epoch: 2 [119554048/128008192 (93%)] Data (t): 0.378 Batch (t): 5.612, 2881.45/s, 180.090/s/gpu LR: 0.000170 Logit Scale: 90.116 Class_loss: 5.1429 (5.2156) Contrastive_loss: 0.68065 (0.70132) Loss: 5.8235 (5.9169)
|
| 472 |
+
2025-05-08,09:28:37 | INFO | Train Epoch: 2 [121651200/128008192 (95%)] Data (t): 0.355 Batch (t): 5.637, 2831.07/s, 176.942/s/gpu LR: 0.000165 Logit Scale: 90.195 Class_loss: 5.1495 (5.2145) Contrastive_loss: 0.64886 (0.70043) Loss: 5.7984 (5.9149)
|
| 473 |
+
2025-05-08,09:36:33 | WARNING | Handling webdataset error (OSError('image file is truncated (96 bytes not processed)')). Ignoring.
|
| 474 |
+
2025-05-08,09:40:36 | INFO | Train Epoch: 2 [123748352/128008192 (97%)] Data (t): 0.385 Batch (t): 5.616, 2873.73/s, 179.608/s/gpu LR: 0.000161 Logit Scale: 90.365 Class_loss: 5.1091 (5.2127) Contrastive_loss: 0.63148 (0.69928) Loss: 5.7406 (5.9120)
|
| 475 |
+
2025-05-08,09:52:34 | INFO | Train Epoch: 2 [125845504/128008192 (98%)] Data (t): 0.378 Batch (t): 5.610, 2923.71/s, 182.732/s/gpu LR: 0.000156 Logit Scale: 90.481 Class_loss: 5.1154 (5.2111) Contrastive_loss: 0.70372 (0.69935) Loss: 5.8191 (5.9105)
|
| 476 |
+
2025-05-08,09:58:26 | WARNING | Handling webdataset error (OSError('image file is truncated (29 bytes not processed)')). Ignoring.
|
| 477 |
+
2025-05-08,10:04:31 | INFO | Train Epoch: 2 [127942656/128008192 (100%)] Data (t): 0.378 Batch (t): 5.604, 2893.73/s, 180.858/s/gpu LR: 0.000151 Logit Scale: 90.709 Class_loss: 5.1799 (5.2106) Contrastive_loss: 0.67976 (0.69904) Loss: 5.8596 (5.9097)
|
| 478 |
+
2025-05-08,10:04:54 | INFO | Train Epoch: 2 [128008192/128008192 (100%)] Data (t): 0.372 Batch (t): 5.605, 3042.14/s, 190.134/s/gpu LR: 0.000151 Logit Scale: 90.710 Class_loss: 5.1339 (5.2094) Contrastive_loss: 0.69610 (0.69899) Loss: 5.8300 (5.9084)
|
| 479 |
+
2025-05-08,10:05:07 | INFO | Start epoch 3
|
| 480 |
+
2025-05-08,10:05:19 | INFO | Train Epoch: 3 [ 16384/128008192 (0%)] Data (t): 7.446 Batch (t): 11.917, 1374.85/s, 85.9284/s/gpu LR: 0.000151 Logit Scale: 90.712 Class_loss: 5.1366 (5.1366) Contrastive_loss: 0.63855 (0.63855) Loss: 5.7751 (5.7751)
|
| 481 |
+
2025-05-08,10:15:21 | WARNING | Handling webdataset error (OSError('image file is truncated (50 bytes not processed)')). Ignoring.
|
| 482 |
+
2025-05-08,10:17:16 | INFO | Train Epoch: 3 [ 2113536/128008192 (2%)] Data (t): 0.405 Batch (t): 5.602, 2897.54/s, 181.096/s/gpu LR: 0.000146 Logit Scale: 90.983 Class_loss: 5.1766 (5.1566) Contrastive_loss: 0.48653 (0.56254) Loss: 5.6632 (5.7191)
|
| 483 |
+
2025-05-08,10:29:17 | INFO | Train Epoch: 3 [ 4210688/128008192 (3%)] Data (t): 0.380 Batch (t): 5.628, 2881.86/s, 180.117/s/gpu LR: 0.000142 Logit Scale: 91.212 Class_loss: 5.1051 (5.1394) Contrastive_loss: 0.59666 (0.57391) Loss: 5.7017 (5.7133)
|
| 484 |
+
2025-05-08,10:41:14 | INFO | Train Epoch: 3 [ 6307840/128008192 (5%)] Data (t): 0.380 Batch (t): 5.608, 2893.25/s, 180.828/s/gpu LR: 0.000137 Logit Scale: 91.416 Class_loss: 5.0294 (5.1119) Contrastive_loss: 0.77461 (0.62409) Loss: 5.8040 (5.7360)
|
| 485 |
+
2025-05-08,10:52:02 | WARNING | Handling webdataset error (OSError('image file is truncated (26 bytes not processed)')). Ignoring.
|
| 486 |
+
2025-05-08,10:53:13 | INFO | Train Epoch: 3 [ 8404992/128008192 (7%)] Data (t): 0.377 Batch (t): 5.614, 2924.65/s, 182.791/s/gpu LR: 0.000133 Logit Scale: 91.553 Class_loss: 5.1105 (5.1116) Contrastive_loss: 0.59555 (0.61838) Loss: 5.7060 (5.7300)
|
| 487 |
+
2025-05-08,11:05:10 | INFO | Train Epoch: 3 [ 10502144/128008192 (8%)] Data (t): 0.381 Batch (t): 5.605, 2922.55/s, 182.659/s/gpu LR: 0.000128 Logit Scale: 91.696 Class_loss: 5.0646 (5.1038) Contrastive_loss: 0.67868 (0.62843) Loss: 5.7433 (5.7322)
|
| 488 |
+
2025-05-08,11:11:13 | WARNING | Handling webdataset error (OSError('image file is truncated (2 bytes not processed)')). Ignoring.
|
| 489 |
+
2025-05-08,11:17:10 | INFO | Train Epoch: 3 [ 12599296/128008192 (10%)] Data (t): 0.382 Batch (t): 5.622, 2957.20/s, 184.825/s/gpu LR: 0.000124 Logit Scale: 91.888 Class_loss: 5.0882 (5.1016) Contrastive_loss: 0.65045 (0.63158) Loss: 5.7387 (5.7332)
|
| 490 |
+
2025-05-08,11:26:27 | WARNING | Handling webdataset error (OSError('broken data stream when reading image file')). Ignoring.
|
| 491 |
+
2025-05-08,11:29:09 | INFO | Train Epoch: 3 [ 14696448/128008192 (11%)] Data (t): 0.374 Batch (t): 5.618, 2966.45/s, 185.403/s/gpu LR: 0.000120 Logit Scale: 92.070 Class_loss: 5.1282 (5.1049) Contrastive_loss: 0.56537 (0.62330) Loss: 5.6936 (5.7282)
|
| 492 |
+
2025-05-08,11:41:04 | INFO | Train Epoch: 3 [ 16793600/128008192 (13%)] Data (t): 0.414 Batch (t): 5.588, 2916.59/s, 182.287/s/gpu LR: 0.000116 Logit Scale: 92.234 Class_loss: 5.1132 (5.1058) Contrastive_loss: 0.68119 (0.62973) Loss: 5.7943 (5.7356)
|
| 493 |
+
2025-05-08,11:49:27 | WARNING | Handling webdataset error (OSError('image file is truncated (88 bytes not processed)')). Ignoring.
|
| 494 |
+
2025-05-08,11:53:05 | INFO | Train Epoch: 3 [ 18890752/128008192 (15%)] Data (t): 0.372 Batch (t): 5.628, 2924.07/s, 182.754/s/gpu LR: 0.000111 Logit Scale: 92.292 Class_loss: 5.0917 (5.1044) Contrastive_loss: 0.69644 (0.63640) Loss: 5.7881 (5.7408)
|
| 495 |
+
2025-05-08,12:05:02 | INFO | Train Epoch: 3 [ 20987904/128008192 (16%)] Data (t): 0.385 Batch (t): 5.601, 2996.44/s, 187.277/s/gpu LR: 0.000107 Logit Scale: 92.515 Class_loss: 5.0724 (5.1015) Contrastive_loss: 0.61401 (0.63437) Loss: 5.6864 (5.7359)
|
| 496 |
+
2025-05-08,12:17:01 | INFO | Train Epoch: 3 [ 23085056/128008192 (18%)] Data (t): 0.375 Batch (t): 5.621, 2967.28/s, 185.455/s/gpu LR: 0.000103 Logit Scale: 92.685 Class_loss: 5.1035 (5.1017) Contrastive_loss: 0.54618 (0.62702) Loss: 5.6497 (5.7287)
|
| 497 |
+
2025-05-08,12:29:00 | INFO | Train Epoch: 3 [ 25182208/128008192 (20%)] Data (t): 0.373 Batch (t): 5.615, 2750.91/s, 171.932/s/gpu LR: 0.000099 Logit Scale: 92.888 Class_loss: 5.1146 (5.1027) Contrastive_loss: 0.63919 (0.62795) Loss: 5.7538 (5.7306)
|
| 498 |
+
2025-05-08,12:34:52 | WARNING | Handling webdataset error (OSError('image file is truncated (10 bytes not processed)')). Ignoring.
|
| 499 |
+
2025-05-08,12:41:06 | INFO | Train Epoch: 3 [ 27279360/128008192 (21%)] Data (t): 0.356 Batch (t): 5.673, 2962.27/s, 185.142/s/gpu LR: 0.000095 Logit Scale: 93.022 Class_loss: 5.1506 (5.1061) Contrastive_loss: 0.51466 (0.61986) Loss: 5.6653 (5.7259)
|
| 500 |
+
2025-05-08,12:51:52 | WARNING | Handling webdataset error (OSError('image file is truncated (12 bytes not processed)')). Ignoring.
|
| 501 |
+
2025-05-08,12:52:59 | INFO | Train Epoch: 3 [ 29376512/128008192 (23%)] Data (t): 0.347 Batch (t): 5.573, 2960.97/s, 185.060/s/gpu LR: 0.000092 Logit Scale: 93.181 Class_loss: 5.0590 (5.1029) Contrastive_loss: 0.78854 (0.63111) Loss: 5.8475 (5.7340)
|
| 502 |
+
2025-05-08,12:58:19 | WARNING | Handling webdataset error (OSError('image file is truncated (9 bytes not processed)')). Ignoring.
|
| 503 |
+
2025-05-08,13:04:55 | INFO | Train Epoch: 3 [ 31473664/128008192 (25%)] Data (t): 0.353 Batch (t): 5.590, 2956.37/s, 184.773/s/gpu LR: 0.000088 Logit Scale: 93.329 Class_loss: 5.1016 (5.1029) Contrastive_loss: 0.59272 (0.62871) Loss: 5.6943 (5.7316)
|
| 504 |
+
2025-05-08,13:04:56 | WARNING | Handling webdataset error (OSError('image file is truncated (4 bytes not processed)')). Ignoring.
|
| 505 |
+
2025-05-08,13:05:06 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
|
| 506 |
+
2025-05-08,13:16:53 | INFO | Train Epoch: 3 [ 33570816/128008192 (26%)] Data (t): 0.380 Batch (t): 5.607, 2925.64/s, 182.853/s/gpu LR: 0.000084 Logit Scale: 93.488 Class_loss: 5.1314 (5.1045) Contrastive_loss: 0.62311 (0.62838) Loss: 5.7545 (5.7329)
|
| 507 |
+
2025-05-08,13:21:59 | WARNING | Handling webdataset error (OSError('image file is truncated (66 bytes not processed)')). Ignoring.
|
| 508 |
+
2025-05-08,13:28:50 | INFO | Train Epoch: 3 [ 35667968/128008192 (28%)] Data (t): 0.373 Batch (t): 5.601, 2924.57/s, 182.786/s/gpu LR: 0.000081 Logit Scale: 93.671 Class_loss: 5.0504 (5.1015) Contrastive_loss: 0.65770 (0.63001) Loss: 5.7081 (5.7315)
|
| 509 |
+
2025-05-08,13:40:48 | INFO | Train Epoch: 3 [ 37765120/128008192 (30%)] Data (t): 0.381 Batch (t): 5.611, 2910.41/s, 181.901/s/gpu LR: 0.000077 Logit Scale: 93.830 Class_loss: 5.1025 (5.1016) Contrastive_loss: 0.72113 (0.63480) Loss: 5.8236 (5.7364)
|
| 510 |
+
2025-05-08,13:52:47 | INFO | Train Epoch: 3 [ 39862272/128008192 (31%)] Data (t): 0.384 Batch (t): 5.622, 2895.21/s, 180.951/s/gpu LR: 0.000074 Logit Scale: 93.990 Class_loss: 5.0440 (5.0987) Contrastive_loss: 0.60184 (0.63316) Loss: 5.6458 (5.7319)
|
| 511 |
+
2025-05-08,14:01:56 | WARNING | Handling webdataset error (OSError('image file is truncated (20 bytes not processed)')). Ignoring.
|
| 512 |
+
2025-05-08,14:04:53 | INFO | Train Epoch: 3 [ 41959424/128008192 (33%)] Data (t): 0.386 Batch (t): 5.671, 2892.17/s, 180.760/s/gpu LR: 0.000070 Logit Scale: 94.133 Class_loss: 5.0615 (5.0969) Contrastive_loss: 0.58736 (0.63097) Loss: 5.6488 (5.7279)
|
| 513 |
+
2025-05-08,14:06:25 | WARNING | Handling webdataset error (OSError('image file is truncated (54 bytes not processed)')). Ignoring.
|
| 514 |
+
2025-05-08,14:16:59 | INFO | Train Epoch: 3 [ 44056576/128008192 (34%)] Data (t): 0.385 Batch (t): 5.666, 2870.14/s, 179.384/s/gpu LR: 0.000067 Logit Scale: 94.280 Class_loss: 4.9797 (5.0916) Contrastive_loss: 0.71320 (0.63471) Loss: 5.6929 (5.7263)
|
| 515 |
+
2025-05-08,14:28:05 | WARNING | Handling webdataset error (OSError('image file is truncated (48 bytes not processed)')). Ignoring.
|
| 516 |
+
2025-05-08,14:29:00 | WARNING | Handling webdataset error (OSError('image file is truncated (18 bytes not processed)')). Ignoring.
|
| 517 |
+
2025-05-08,14:29:03 | INFO | Train Epoch: 3 [ 46153728/128008192 (36%)] Data (t): 0.379 Batch (t): 5.659, 2949.66/s, 184.354/s/gpu LR: 0.000064 Logit Scale: 94.417 Class_loss: 5.0640 (5.0904) Contrastive_loss: 0.68435 (0.63687) Loss: 5.7483 (5.7273)
|
| 518 |
+
2025-05-08,14:41:06 | INFO | Train Epoch: 3 [ 48250880/128008192 (38%)] Data (t): 0.384 Batch (t): 5.648, 2877.27/s, 179.830/s/gpu LR: 0.000061 Logit Scale: 94.524 Class_loss: 5.0510 (5.0888) Contrastive_loss: 0.63125 (0.63664) Loss: 5.6822 (5.7254)
|
| 519 |
+
2025-05-08,14:51:54 | WARNING | Handling webdataset error (OSError('image file is truncated (152 bytes not processed)')). Ignoring.
|
| 520 |
+
2025-05-08,14:53:09 | INFO | Train Epoch: 3 [ 50348032/128008192 (39%)] Data (t): 0.383 Batch (t): 5.650, 2920.36/s, 182.522/s/gpu LR: 0.000058 Logit Scale: 94.686 Class_loss: 5.0669 (5.0879) Contrastive_loss: 0.65117 (0.63722) Loss: 5.7181 (5.7251)
|
| 521 |
+
2025-05-08,15:01:33 | WARNING | Handling webdataset error (OSError('image file is truncated (22 bytes not processed)')). Ignoring.
|
| 522 |
+
2025-05-08,15:05:16 | INFO | Train Epoch: 3 [ 52445184/128008192 (41%)] Data (t): 0.384 Batch (t): 5.678, 2933.26/s, 183.329/s/gpu LR: 0.000055 Logit Scale: 94.804 Class_loss: 5.0312 (5.0857) Contrastive_loss: 0.62053 (0.63658) Loss: 5.6517 (5.7223)
|
| 523 |
+
2025-05-08,15:17:24 | INFO | Train Epoch: 3 [ 54542336/128008192 (43%)] Data (t): 0.430 Batch (t): 5.686, 2832.42/s, 177.026/s/gpu LR: 0.000052 Logit Scale: 94.913 Class_loss: 5.0348 (5.0838) Contrastive_loss: 0.73634 (0.64027) Loss: 5.7711 (5.7241)
|
| 524 |
+
2025-05-08,15:24:12 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
|
| 525 |
+
2025-05-08,15:29:29 | INFO | Train Epoch: 3 [ 56639488/128008192 (44%)] Data (t): 0.381 Batch (t): 5.668, 2892.29/s, 180.768/s/gpu LR: 0.000049 Logit Scale: 95.057 Class_loss: 5.0671 (5.0832) Contrastive_loss: 0.55752 (0.63732) Loss: 5.6246 (5.7205)
|
| 526 |
+
2025-05-08,15:41:07 | WARNING | Handling webdataset error (OSError('image file is truncated (2 bytes not processed)')). Ignoring.
|
| 527 |
+
2025-05-08,15:41:34 | INFO | Train Epoch: 3 [ 58736640/128008192 (46%)] Data (t): 0.384 Batch (t): 5.658, 2895.38/s, 180.961/s/gpu LR: 0.000046 Logit Scale: 95.194 Class_loss: 5.0545 (5.0822) Contrastive_loss: 0.55204 (0.63437) Loss: 5.6065 (5.7166)
|
| 528 |
+
2025-05-08,15:47:34 | WARNING | Handling webdataset error (OSError('image file is truncated (31 bytes not processed)')). Ignoring.
|
| 529 |
+
2025-05-08,15:53:40 | INFO | Train Epoch: 3 [ 60833792/128008192 (48%)] Data (t): 0.387 Batch (t): 5.676, 2836.01/s, 177.250/s/gpu LR: 0.000043 Logit Scale: 95.314 Class_loss: 5.0949 (5.0826) Contrastive_loss: 0.49555 (0.62975) Loss: 5.5905 (5.7124)
|
| 530 |
+
2025-05-08,16:05:47 | INFO | Train Epoch: 3 [ 62930944/128008192 (49%)] Data (t): 0.387 Batch (t): 5.682, 2877.59/s, 179.849/s/gpu LR: 0.000041 Logit Scale: 95.423 Class_loss: 5.0718 (5.0823) Contrastive_loss: 0.70084 (0.63204) Loss: 5.7727 (5.7143)
|
| 531 |
+
2025-05-08,16:17:54 | INFO | Train Epoch: 3 [ 65028096/128008192 (51%)] Data (t): 0.389 Batch (t): 5.679, 2930.45/s, 183.153/s/gpu LR: 0.000038 Logit Scale: 95.546 Class_loss: 5.0884 (5.0825) Contrastive_loss: 0.59269 (0.63081) Loss: 5.6811 (5.7133)
|
| 532 |
+
2025-05-08,16:22:23 | WARNING | Handling webdataset error (OSError('image file is truncated (92 bytes not processed)')). Ignoring.
|
| 533 |
+
2025-05-08,16:29:59 | INFO | Train Epoch: 3 [ 67125248/128008192 (52%)] Data (t): 0.384 Batch (t): 5.659, 2921.82/s, 182.614/s/gpu LR: 0.000036 Logit Scale: 95.641 Class_loss: 5.0605 (5.0818) Contrastive_loss: 0.60626 (0.63007) Loss: 5.6668 (5.7119)
|
| 534 |
+
2025-05-08,16:32:52 | WARNING | Handling webdataset error (OSError('image file is truncated (13 bytes not processed)')). Ignoring.
|
| 535 |
+
2025-05-08,16:42:03 | INFO | Train Epoch: 3 [ 69222400/128008192 (54%)] Data (t): 0.384 Batch (t): 5.657, 2892.37/s, 180.773/s/gpu LR: 0.000033 Logit Scale: 95.751 Class_loss: 5.0640 (5.0813) Contrastive_loss: 0.73074 (0.63303) Loss: 5.7948 (5.7143)
|
| 536 |
+
2025-05-08,16:45:15 | WARNING | Handling webdataset error (OSError('image file is truncated (123 bytes not processed)')). Ignoring.
|
| 537 |
+
2025-05-08,16:54:11 | INFO | Train Epoch: 3 [ 71319552/128008192 (56%)] Data (t): 0.384 Batch (t): 5.687, 2852.71/s, 178.295/s/gpu LR: 0.000031 Logit Scale: 95.846 Class_loss: 5.0269 (5.0797) Contrastive_loss: 0.78363 (0.63733) Loss: 5.8105 (5.7171)
|
| 538 |
+
2025-05-08,17:06:17 | INFO | Train Epoch: 3 [ 73416704/128008192 (57%)] Data (t): 0.387 Batch (t): 5.673, 2899.51/s, 181.219/s/gpu LR: 0.000029 Logit Scale: 95.931 Class_loss: 5.0728 (5.0796) Contrastive_loss: 0.64921 (0.63766) Loss: 5.7220 (5.7172)
|
| 539 |
+
2025-05-08,17:18:20 | INFO | Train Epoch: 3 [ 75513856/128008192 (59%)] Data (t): 0.380 Batch (t): 5.647, 2875.99/s, 179.750/s/gpu LR: 0.000027 Logit Scale: 96.028 Class_loss: 5.0836 (5.0797) Contrastive_loss: 0.71288 (0.63969) Loss: 5.7965 (5.7194)
|
| 540 |
+
2025-05-08,17:24:16 | WARNING | Handling webdataset error (OSError('image file is truncated (27 bytes not processed)')). Ignoring.
|
| 541 |
+
2025-05-08,17:30:23 | INFO | Train Epoch: 3 [ 77611008/128008192 (61%)] Data (t): 0.385 Batch (t): 5.648, 2893.80/s, 180.863/s/gpu LR: 0.000025 Logit Scale: 96.106 Class_loss: 5.0521 (5.0789) Contrastive_loss: 0.61884 (0.63914) Loss: 5.6710 (5.7181)
|
| 542 |
+
2025-05-08,17:40:09 | WARNING | Handling webdataset error (OSError('image file is truncated (74 bytes not processed)')). Ignoring.
|
| 543 |
+
2025-05-08,17:42:23 | INFO | Train Epoch: 3 [ 79708160/128008192 (62%)] Data (t): 0.384 Batch (t): 5.632, 2921.75/s, 182.610/s/gpu LR: 0.000023 Logit Scale: 96.197 Class_loss: 5.0220 (5.0775) Contrastive_loss: 0.79093 (0.64304) Loss: 5.8129 (5.7205)
|
| 544 |
+
2025-05-08,17:54:29 | INFO | Train Epoch: 3 [ 81805312/128008192 (64%)] Data (t): 0.369 Batch (t): 5.670, 2859.62/s, 178.726/s/gpu LR: 0.000021 Logit Scale: 96.289 Class_loss: 5.0475 (5.0767) Contrastive_loss: 0.59698 (0.64189) Loss: 5.6445 (5.7186)
|
| 545 |
+
2025-05-08,17:58:43 | WARNING | Handling webdataset error (OSError('image file is truncated (2 bytes not processed)')). Ignoring.
|
| 546 |
+
2025-05-08,18:05:00 | WARNING | Handling webdataset error (OSError('image file is truncated (7 bytes not processed)')). Ignoring.
|
| 547 |
+
2025-05-08,18:06:43 | INFO | Train Epoch: 3 [ 83902464/128008192 (66%)] Data (t): 0.387 Batch (t): 5.734, 2892.23/s, 180.764/s/gpu LR: 0.000019 Logit Scale: 96.380 Class_loss: 4.9776 (5.0743) Contrastive_loss: 0.73529 (0.64416) Loss: 5.7129 (5.7185)
|
| 548 |
+
2025-05-08,18:15:23 | WARNING | Handling webdataset error (OSError('image file is truncated (92 bytes not processed)')). Ignoring.
|
| 549 |
+
2025-05-08,18:18:51 | INFO | Train Epoch: 3 [ 85999616/128008192 (67%)] Data (t): 0.382 Batch (t): 5.683, 2803.68/s, 175.230/s/gpu LR: 0.000017 Logit Scale: 96.452 Class_loss: 5.0471 (5.0737) Contrastive_loss: 0.56527 (0.64229) Loss: 5.6124 (5.7159)
|
| 550 |
+
2025-05-08,18:30:51 | INFO | Train Epoch: 3 [ 88096768/128008192 (69%)] Data (t): 0.366 Batch (t): 5.627, 2849.40/s, 178.087/s/gpu LR: 0.000015 Logit Scale: 96.506 Class_loss: 5.1201 (5.0747) Contrastive_loss: 0.56000 (0.64037) Loss: 5.6801 (5.7151)
|
| 551 |
+
2025-05-08,18:33:22 | WARNING | Handling webdataset error (OSError('image file is truncated (121 bytes not processed)')). Ignoring.
|
| 552 |
+
2025-05-08,18:37:59 | WARNING | Handling webdataset error (OSError('image file is truncated (80 bytes not processed)')). Ignoring.
|
| 553 |
+
2025-05-08,18:38:36 | WARNING | Handling webdataset error (OSError('image file is truncated (58 bytes not processed)')). Ignoring.
|
| 554 |
+
2025-05-08,18:42:57 | INFO | Train Epoch: 3 [ 90193920/128008192 (70%)] Data (t): 0.385 Batch (t): 5.671, 2885.22/s, 180.327/s/gpu LR: 0.000014 Logit Scale: 96.564 Class_loss: 5.0106 (5.0733) Contrastive_loss: 0.60482 (0.63956) Loss: 5.6154 (5.7128)
|
| 555 |
+
2025-05-08,18:43:28 | WARNING | Handling webdataset error (OSError('image file is truncated (18 bytes not processed)')). Ignoring.
|
| 556 |
+
2025-05-08,18:55:02 | INFO | Train Epoch: 3 [ 92291072/128008192 (72%)] Data (t): 0.383 Batch (t): 5.667, 3042.79/s, 190.174/s/gpu LR: 0.000012 Logit Scale: 96.618 Class_loss: 5.0731 (5.0733) Contrastive_loss: 0.61289 (0.63897) Loss: 5.6860 (5.7123)
|
| 557 |
+
2025-05-08,19:07:07 | INFO | Train Epoch: 3 [ 94388224/128008192 (74%)] Data (t): 0.389 Batch (t): 5.662, 2911.67/s, 181.979/s/gpu LR: 0.000011 Logit Scale: 96.656 Class_loss: 5.0052 (5.0718) Contrastive_loss: 0.66912 (0.63963) Loss: 5.6743 (5.7114)
|
| 558 |
+
2025-05-08,19:12:40 | WARNING | Handling webdataset error (OSError('image file is truncated (24 bytes not processed)')). Ignoring.
|
| 559 |
+
2025-05-08,19:19:04 | INFO | Train Epoch: 3 [ 96485376/128008192 (75%)] Data (t): 0.378 Batch (t): 5.606, 2927.70/s, 182.981/s/gpu LR: 0.000010 Logit Scale: 96.703 Class_loss: 5.0235 (5.0708) Contrastive_loss: 0.69835 (0.64088) Loss: 5.7219 (5.7116)
|
| 560 |
+
2025-05-08,19:20:30 | WARNING | Handling webdataset error (OSError('image file is truncated (86 bytes not processed)')). Ignoring.
|
| 561 |
+
2025-05-08,19:31:08 | INFO | Train Epoch: 3 [ 98582528/128008192 (77%)] Data (t): 0.386 Batch (t): 5.657, 2857.66/s, 178.604/s/gpu LR: 0.000008 Logit Scale: 96.745 Class_loss: 5.0406 (5.0701) Contrastive_loss: 0.65326 (0.64113) Loss: 5.6939 (5.7113)
|
| 562 |
+
2025-05-08,19:43:11 | INFO | Train Epoch: 3 [100679680/128008192 (79%)] Data (t): 0.384 Batch (t): 5.646, 2886.31/s, 180.394/s/gpu LR: 0.000007 Logit Scale: 96.780 Class_loss: 5.0539 (5.0698) Contrastive_loss: 0.49114 (0.63807) Loss: 5.5450 (5.7079)
|
| 563 |
+
2025-05-08,19:50:36 | WARNING | Handling webdataset error (OSError('image file is truncated (26 bytes not processed)')). Ignoring.
|
| 564 |
+
2025-05-08,19:55:13 | INFO | Train Epoch: 3 [102776832/128008192 (80%)] Data (t): 0.373 Batch (t): 5.641, 2929.06/s, 183.066/s/gpu LR: 0.000006 Logit Scale: 96.805 Class_loss: 5.0424 (5.0693) Contrastive_loss: 0.69789 (0.63927) Loss: 5.7403 (5.7085)
|
| 565 |
+
2025-05-08,19:58:17 | WARNING | Handling webdataset error (OSError('image file is truncated (97 bytes not processed)')). Ignoring.
|
| 566 |
+
2025-05-08,20:07:10 | INFO | Train Epoch: 3 [104873984/128008192 (82%)] Data (t): 0.376 Batch (t): 5.598, 2789.93/s, 174.371/s/gpu LR: 0.000005 Logit Scale: 96.833 Class_loss: 5.0545 (5.0690) Contrastive_loss: 0.67182 (0.63991) Loss: 5.7263 (5.7089)
|
| 567 |
+
2025-05-08,20:19:13 | INFO | Train Epoch: 3 [106971136/128008192 (84%)] Data (t): 0.385 Batch (t): 5.653, 2880.93/s, 180.058/s/gpu LR: 0.000004 Logit Scale: 96.847 Class_loss: 5.0025 (5.0677) Contrastive_loss: 0.62993 (0.63972) Loss: 5.6325 (5.7074)
|
| 568 |
+
2025-05-08,20:19:34 | WARNING | Handling webdataset error (OSError('broken data stream when reading image file')). Ignoring.
|
| 569 |
+
2025-05-08,20:31:20 | INFO | Train Epoch: 3 [109068288/128008192 (85%)] Data (t): 0.387 Batch (t): 5.680, 2848.18/s, 178.011/s/gpu LR: 0.000003 Logit Scale: 96.866 Class_loss: 5.0975 (5.0683) Contrastive_loss: 0.57457 (0.63849) Loss: 5.6721 (5.7067)
|
| 570 |
+
2025-05-08,20:34:57 | WARNING | Handling webdataset error (OSError('image file is truncated (28 bytes not processed)')). Ignoring.
|
| 571 |
+
2025-05-08,20:43:27 | INFO | Train Epoch: 3 [111165440/128008192 (87%)] Data (t): 0.386 Batch (t): 5.676, 2946.26/s, 184.141/s/gpu LR: 0.000003 Logit Scale: 96.881 Class_loss: 5.0587 (5.0681) Contrastive_loss: 0.56383 (0.63710) Loss: 5.6225 (5.7052)
|
| 572 |
+
2025-05-08,20:46:13 | WARNING | Handling webdataset error (OSError('image file is truncated (58 bytes not processed)')). Ignoring.
|
| 573 |
+
2025-05-08,20:55:32 | INFO | Train Epoch: 3 [113262592/128008192 (88%)] Data (t): 0.385 Batch (t): 5.662, 2907.24/s, 181.703/s/gpu LR: 0.000002 Logit Scale: 96.888 Class_loss: 5.0340 (5.0675) Contrastive_loss: 0.67713 (0.63783) Loss: 5.7112 (5.7053)
|
| 574 |
+
2025-05-08,20:59:01 | WARNING | Handling webdataset error (OSError('image file is truncated (17 bytes not processed)')). Ignoring.
|
| 575 |
+
2025-05-08,21:07:37 | INFO | Train Epoch: 3 [115359744/128008192 (90%)] Data (t): 0.382 Batch (t): 5.667, 2948.27/s, 184.267/s/gpu LR: 0.000002 Logit Scale: 96.894 Class_loss: 5.0026 (5.0663) Contrastive_loss: 0.70279 (0.63899) Loss: 5.7054 (5.7053)
|
| 576 |
+
2025-05-08,21:12:09 | WARNING | Handling webdataset error (OSError('image file is truncated (6 bytes not processed)')). Ignoring.
|
| 577 |
+
2025-05-08,21:19:43 | INFO | Train Epoch: 3 [117456896/128008192 (92%)] Data (t): 0.420 Batch (t): 5.677, 2819.59/s, 176.224/s/gpu LR: 0.000001 Logit Scale: 96.897 Class_loss: 5.0220 (5.0655) Contrastive_loss: 0.62688 (0.63878) Loss: 5.6488 (5.7043)
|
| 578 |
+
2025-05-08,21:27:58 | WARNING | Handling webdataset error (OSError('image file is truncated (37 bytes not processed)')). Ignoring.
|
| 579 |
+
2025-05-08,21:31:49 | INFO | Train Epoch: 3 [119554048/128008192 (93%)] Data (t): 0.389 Batch (t): 5.669, 2868.55/s, 179.285/s/gpu LR: 0.000001 Logit Scale: 96.900 Class_loss: 5.0118 (5.0646) Contrastive_loss: 0.60763 (0.63824) Loss: 5.6194 (5.7028)
|
| 580 |
+
2025-05-08,21:33:14 | WARNING | Handling webdataset error (OSError('image file is truncated (70 bytes not processed)')). Ignoring.
|
| 581 |
+
2025-05-08,21:43:53 | INFO | Train Epoch: 3 [121651200/128008192 (95%)] Data (t): 0.382 Batch (t): 5.657, 2884.05/s, 180.253/s/gpu LR: 0.000000 Logit Scale: 96.901 Class_loss: 5.0075 (5.0636) Contrastive_loss: 0.63033 (0.63811) Loss: 5.6378 (5.7017)
|
| 582 |
+
2025-05-08,21:56:05 | INFO | Train Epoch: 3 [123748352/128008192 (97%)] Data (t): 0.383 Batch (t): 5.721, 2886.43/s, 180.402/s/gpu LR: 0.000000 Logit Scale: 96.901 Class_loss: 5.0342 (5.0631) Contrastive_loss: 0.68701 (0.63892) Loss: 5.7212 (5.7021)
|
| 583 |
+
2025-05-08,21:57:39 | WARNING | Handling webdataset error (OSError('image file is truncated (31 bytes not processed)')). Ignoring.
|
| 584 |
+
2025-05-08,22:08:05 | INFO | Train Epoch: 3 [125845504/128008192 (98%)] Data (t): 0.375 Batch (t): 5.621, 2880.87/s, 180.055/s/gpu LR: 0.000000 Logit Scale: 96.901 Class_loss: 5.0535 (5.0630) Contrastive_loss: 0.57365 (0.63785) Loss: 5.6272 (5.7008)
|
| 585 |
+
2025-05-08,22:20:15 | INFO | Train Epoch: 3 [127942656/128008192 (100%)] Data (t): 0.382 Batch (t): 5.706, 2978.99/s, 186.187/s/gpu LR: 0.000000 Logit Scale: 96.901 Class_loss: 5.0401 (5.0626) Contrastive_loss: 0.68409 (0.63860) Loss: 5.7241 (5.7012)
|
| 586 |
+
2025-05-08,22:20:37 | INFO | Train Epoch: 3 [128008192/128008192 (100%)] Data (t): 0.387 Batch (t): 5.567, 3066.85/s, 191.678/s/gpu LR: 0.000000 Logit Scale: 96.901 Class_loss: 5.0807 (5.0629) Contrastive_loss: 0.59234 (0.63786) Loss: 5.6730 (5.7008)
|
| 587 |
+
2025-05-08,22:20:44 | INFO | Starting zero-shot imagenet.
|
| 588 |
+
2025-05-08,22:20:44 | INFO | Building zero-shot classifier
|
| 589 |
+
2025-05-08,22:21:02 | INFO | Using classifier
|
clipcls_vit_b16_s512m_bs16k_mix0_0/params.txt
ADDED
|
@@ -0,0 +1,109 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
NDR_patch_size: 16
|
| 2 |
+
accum_freq: 1
|
| 3 |
+
aug_cfg: {}
|
| 4 |
+
batch_size: 1024
|
| 5 |
+
beta1: 0.9
|
| 6 |
+
beta2: 0.98
|
| 7 |
+
checkpoint_path: ./logs-lr1e-3-datacomp/clipcls_vit_b16_s512m_bs16k_mix0_0/checkpoints
|
| 8 |
+
coca_caption_loss_weight: 2.0
|
| 9 |
+
coca_contrastive_loss_weight: 1.0
|
| 10 |
+
copy_codebase: False
|
| 11 |
+
csv_caption_key: title
|
| 12 |
+
csv_img_key: filepath
|
| 13 |
+
csv_separator:
|
| 14 |
+
dataset_resampled: False
|
| 15 |
+
dataset_type: webdataset
|
| 16 |
+
ddp_static_graph: True
|
| 17 |
+
debug: False
|
| 18 |
+
delete_prev_step_ckpt: True
|
| 19 |
+
delete_previous_checkpoint: False
|
| 20 |
+
device: cuda:0
|
| 21 |
+
dist_backend: nccl
|
| 22 |
+
dist_url: env://
|
| 23 |
+
distill: False
|
| 24 |
+
distill_model: None
|
| 25 |
+
distill_pretrained: None
|
| 26 |
+
distributed: True
|
| 27 |
+
epochs: 4
|
| 28 |
+
epochs_cooldown: None
|
| 29 |
+
eps: 1e-06
|
| 30 |
+
force_custom_text: False
|
| 31 |
+
force_image_size: 224
|
| 32 |
+
force_patch_dropout: None
|
| 33 |
+
force_quick_gelu: False
|
| 34 |
+
gather_with_grad: True
|
| 35 |
+
global_batch_size: 16384
|
| 36 |
+
grad_checkpointing: True
|
| 37 |
+
grad_clip_norm: None
|
| 38 |
+
horovod: False
|
| 39 |
+
image_interpolation: None
|
| 40 |
+
image_mean: None
|
| 41 |
+
image_resize_mode: None
|
| 42 |
+
image_std: None
|
| 43 |
+
imagenet_v2: None
|
| 44 |
+
imagenet_val: /mnt/bn/zilongdata-hl/dataset/imagenet/val
|
| 45 |
+
is_cls_token: True
|
| 46 |
+
local_loss: True
|
| 47 |
+
local_rank: 0
|
| 48 |
+
lock_image: False
|
| 49 |
+
lock_image_freeze_bn_stats: False
|
| 50 |
+
lock_image_unlocked_groups: 0
|
| 51 |
+
lock_text: False
|
| 52 |
+
lock_text_freeze_layer_norm: False
|
| 53 |
+
lock_text_unlocked_layers: 0
|
| 54 |
+
log_every_n_steps: 128
|
| 55 |
+
log_level: 20
|
| 56 |
+
log_local: False
|
| 57 |
+
log_path: ./logs-lr1e-3-datacomp/clipcls_vit_b16_s512m_bs16k_mix0_0/out.log
|
| 58 |
+
logs: ./logs-lr1e-3-datacomp
|
| 59 |
+
lr: 0.001
|
| 60 |
+
lr_cooldown_end: 0.0
|
| 61 |
+
lr_cooldown_power: 1.0
|
| 62 |
+
lr_scheduler: cosine
|
| 63 |
+
max_seq_len: 15000
|
| 64 |
+
model: CLIPCLS-ViT-B-16
|
| 65 |
+
name: clipcls_vit_b16_s512m_bs16k_mix0_0
|
| 66 |
+
native_dynamic_resolution: False
|
| 67 |
+
no_set_device_rank: False
|
| 68 |
+
only_packing: False
|
| 69 |
+
precision: amp
|
| 70 |
+
pretrained:
|
| 71 |
+
pretrained_image:
|
| 72 |
+
pretrained_text:
|
| 73 |
+
rank: 0
|
| 74 |
+
remote_sync: None
|
| 75 |
+
remote_sync_frequency: 300
|
| 76 |
+
remote_sync_protocol: s3
|
| 77 |
+
report_to: wandb
|
| 78 |
+
resume: None
|
| 79 |
+
rope_attn_num_heads: 12
|
| 80 |
+
rope_model_width: 768
|
| 81 |
+
save_every_n_steps: 6104
|
| 82 |
+
save_frequency: 1
|
| 83 |
+
save_most_recent: False
|
| 84 |
+
seed: 0
|
| 85 |
+
siglip: False
|
| 86 |
+
skip_scheduler: False
|
| 87 |
+
tensorboard: False
|
| 88 |
+
tensorboard_path:
|
| 89 |
+
torchcompile: False
|
| 90 |
+
torchscript: False
|
| 91 |
+
trace: False
|
| 92 |
+
train_data: /mnt/bn/zilongdata-hl/dataset/Recap-DataComp-1B-Dataset/{000000..140146}.tar
|
| 93 |
+
train_data_upsampling_factors: None
|
| 94 |
+
train_num_samples: 128000000
|
| 95 |
+
use_bn_sync: False
|
| 96 |
+
use_bnb_linear: None
|
| 97 |
+
val_data: None
|
| 98 |
+
val_frequency: 1
|
| 99 |
+
val_num_samples: None
|
| 100 |
+
val_steps: 0
|
| 101 |
+
wandb: True
|
| 102 |
+
wandb_notes:
|
| 103 |
+
wandb_project_name: cls-clip-NDR
|
| 104 |
+
warmup: 500
|
| 105 |
+
wd: 0.2
|
| 106 |
+
workers: 1
|
| 107 |
+
world_size: 16
|
| 108 |
+
zeroshot_frequency: 4
|
| 109 |
+
zeroshot_steps: 0
|