Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I've been Verifying settings while fine-tuning the dataset... After a long time, the cursor kept flashing #1887

Open
PiaoGengHan opened this issue Dec 24, 2024 · 1 comment
Labels
question Further information is requested

Comments

@PiaoGengHan
Copy link

Here I used the dataset given by git as a test for the initial fine-tuning, but there was no response, and then I thought that the dataset might be too large, resulting in too long a time to split the validation set and the training set, I did some datasets according to his format for testing, or stuck in Verifying settings... No response for a long time, here is the log from the command line. I wonder if something went wrong with the settings. Thank you very much for your answers!!

Please describe yourAIGC\litgpt\litgpt>litgpt finetune microsoft/phi-2 --data JSON --data.json_path my_custom_dataset.json --data.val_split_fraction 0.1 --out_dir out/custom-model
{'access_token': None,
'checkpoint_dir': WindowsPath('checkpoints/microsoft/phi-2'),
'data': JSON(json_path=WindowsPath('my_custom_dataset.json'),
mask_prompt=False,
val_split_fraction=0.1,
prompt_style=<litgpt.prompts.Alpaca object at 0x0000023E1F336440>,
ignore_index=-100,
seed=42,
num_workers=4),
'devices': 1,
'eval': EvalArgs(interval=100,
max_new_tokens=100,
max_iters=100,
initial_validation=False,
final_validation=True,
evaluate_example='first'),
'logger_name': 'csv',
'lora_alpha': 16,
'lora_dropout': 0.05,
'lora_head': False,
'lora_key': False,
'lora_mlp': False,
'lora_projection': False,
'lora_query': True,
'lora_r': 8,
'lora_value': True,
'num_nodes': 1,
'optimizer': 'AdamW',
'out_dir': WindowsPath('out/custom-model'),
'precision': None,
'quantize': None,
'seed': 1337,
'train': TrainArgs(save_interval=1000,
log_interval=1,
global_batch_size=16,
micro_batch_size=1,
lr_warmup_steps=100,
lr_warmup_fraction=None,
epochs=5,
max_tokens=None,
max_steps=None,
max_seq_length=None,
tie_embeddings=None,
max_norm=None,
min_lr=6e-05)}
Using bfloat16 Automatic Mixed Precision (AMP)
Seed set to 1337
Number of trainable parameters: 2,621,440
Number of non-trainable parameters: 2,779,683,840
The longest sequence length in the train data is 363, the model's maximum sequence length is 363 and context length is 2048
Verifying settings ...

@PiaoGengHan PiaoGengHan added the question Further information is requested label Dec 24, 2024
@PiaoGengHan
Copy link
Author

Here is the device information:
Device name DESKTOP-VAT73EM
Processor Intel(R) Core(TM) i9-14900KF 3.20 GHz
Band RAM 64.0 GB (63.8 GB available)
Graphics card 4090
System type 64-bit operating system, x64-based processor
Disk 2T SSD + 10T HDD
Tuned model microsoft/phi-2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant