lucataco/internlm2_5-7b-chat – Run with an API on Replicate

lucataco / internlm2_5-7b-chat

InternLM2.5 has open-sourced a 7 billion parameter base model and a chat model tailored for practical scenarios.

Cold

Public
58 runs
L40S
Paper
License

Run with an API

Playground API Examples Train README Versions

Input

prompt

string

Shift + Return to add a new line

please provide three suggestions about time managementplease provide three suggestions about time management

Prompt

Default: ""

system_prompt

string

Shift + Return to add a new line

System prompt to send to the model. This is prepended to the prompt and helps guide system behavior. Ignored for non-chat models.

Default: "You are a helpful assistant."

min_tokens

integer

The minimum number of tokens the model should generate as output.

Default: 0

max_tokens

integer

The maximum number of tokens the model should generate as output.

Default: 512

temperature

number

The value used to modulate the next token probabilities.

Default: 0.6

top_p

number

A probability threshold for generating the output. If < 1.0, only keep the top tokens with cumulative probability >= top_p (nucleus filtering). Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751).

Default: 0.9

top_k

integer

The number of highest probability tokens to consider for generating the output. If > 0, only keep the top k tokens with highest probability (top-k filtering).

Default: 50

presence_penalty

number

Presence penalty

Default: 0

frequency_penalty

number

Frequency penalty

Default: 0

stop_sequences

string

Shift + Return to add a new line

A comma-separated list of sequences to stop generation at. For example, '<end>,<stop>' will stop generation at the first instance of 'end' or '<stop>'.

Run this model in Node.js with one line of code:

npx create-replicate --model=lucataco/internlm2_5-7b-chat

or set up a project from scratch

Install Replicate’s Node.js client library:

npm install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import and set up the client:

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
});

Run lucataco/internlm2_5-7b-chat using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

const output = await replicate.run(
  "lucataco/internlm2_5-7b-chat:e943ac18ac8c05ac7deb01cbc3aba827b74f1ecdcb71d722e692d7ed4e70c33d",
  {
    input: {
      top_k: 50,
      top_p: 0.9,
      prompt: "please provide three suggestions about time management",
      max_tokens: 512,
      min_tokens: 0,
      temperature: 0.6,
      system_prompt: "You are a helpful assistant.",
      presence_penalty: 0,
      frequency_penalty: 0
    }
  }
);

console.log(output);

To learn more, take a look at the guide on getting started with Node.js.

Install Replicate’s Python client library:

pip install replicate

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Import the client:

import replicate

Run lucataco/internlm2_5-7b-chat using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

output = replicate.run(
    "lucataco/internlm2_5-7b-chat:e943ac18ac8c05ac7deb01cbc3aba827b74f1ecdcb71d722e692d7ed4e70c33d",
    input={
        "top_k": 50,
        "top_p": 0.9,
        "prompt": "please provide three suggestions about time management",
        "max_tokens": 512,
        "min_tokens": 0,
        "temperature": 0.6,
        "system_prompt": "You are a helpful assistant.",
        "presence_penalty": 0,
        "frequency_penalty": 0
    }
)

# The lucataco/internlm2_5-7b-chat model can stream output as it's running.
# The predict method returns an iterator, and you can iterate over that output.
for item in output:
    # https://replicate.com/lucataco/internlm2_5-7b-chat/api#output-schema
    print(item, end="")

To learn more, take a look at the guide on getting started with Python.

Set the REPLICATE_API_TOKEN environment variable:

export REPLICATE_API_TOKEN=<paste-your-token-here>

Find your API token in your account settings.

Run lucataco/internlm2_5-7b-chat using Replicate’s API. Check out the model's schema for an overview of inputs and outputs.

curl -s -X POST \
  -H "Authorization: Bearer $REPLICATE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Prefer: wait" \
  -d $'{
    "version": "lucataco/internlm2_5-7b-chat:e943ac18ac8c05ac7deb01cbc3aba827b74f1ecdcb71d722e692d7ed4e70c33d",
    "input": {
      "top_k": 50,
      "top_p": 0.9,
      "prompt": "please provide three suggestions about time management",
      "max_tokens": 512,
      "min_tokens": 0,
      "temperature": 0.6,
      "system_prompt": "You are a helpful assistant.",
      "presence_penalty": 0,
      "frequency_penalty": 0
    }
  }' \
  https://api.replicate.com/v1/predictions

To learn more, take a look at Replicate’s HTTP API reference docs.

Output

Certainly! Here are three suggestions for effective time management: 1. **Prioritize Tasks**: Start by identifying the most important tasks that need to be completed and prioritize them based on urgency and importance. Use tools like the Eisenhower Matrix to categorize tasks into four quadrants: urgent and important, important but not urgent, urgent but not important, and neither urgent nor important. Focus on completing tasks in the first quadrant first, as they contribute the most to your goals. 2. **Set Specific Goals**: Define clear, achievable goals for each day, week, and month. Break down larger projects into smaller, manageable tasks and set deadlines for each. This helps in maintaining focus and provides a sense of accomplishment as you complete each task. 3. **Use Time-Blocking**: Allocate specific blocks of time for different activities throughout your day. This involves scheduling tasks in your calendar, ensuring you dedicate time for work, breaks, and personal activities. Time-blocking helps in minimizing distractions and ensures that you have a structured approach to your day, maximizing productivity and reducing procrastination. Remember, effective time management is about balancing work with rest and leisure, so ensure you schedule time for relaxation and hobbies to maintain a healthy work-life balance.

{
  "completed_at": "2024-07-04T01:14:19.800970Z",
  "created_at": "2024-07-04T01:12:25.716000Z",
  "data_removed": false,
  "error": null,
  "id": "hjke4jdzehrgj0cgfd684mexxg",
  "input": {
    "top_k": 50,
    "top_p": 0.9,
    "prompt": "please provide three suggestions about time management",
    "max_tokens": 512,
    "min_tokens": 0,
    "temperature": 0.6,
    "system_prompt": "You are a helpful assistant.",
    "presence_penalty": 0,
    "frequency_penalty": 0
  },
  "logs": "INFO 07-04 01:14:12 async_llm_engine.py:529] Received request 6fffef91b60745a99f3323d98d0adf10: prompt: '<s><|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\nplease provide three suggestions about time management<|im_end|>\\n<|im_start|>assistant\\n', sampling_params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.9, top_k=50, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[2], include_stop_str_in_output=False, ignore_eos=False, max_tokens=512, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: None, lora_request: None.\n stdoutINFO 07-04 01:14:16 metrics.py:334] Avg prompt throughput: 5.8 tokens/s, Avg generation throughput: 29.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%\n stdoutGeneration took 1720054315.80sINFO 07-04 01:14:19 async_llm_engine.py:120] Finished request 6fffef91b60745a99f3323d98d0adf10.\n stdoutFormatted prompt: <s><|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nplease provide three suggestions about time management<|im_end|>\n<|im_start|>assistant",
  "metrics": {
    "predict_time": 6.915671904,
    "total_time": 114.08497
  },
  "output": [
    "C",
    "ertain",
    "ly",
    "!",
    " Here",
    " are",
    " three",
    " suggestions",
    " for",
    " effective",
    " time",
    " management",
    ":\n\n",
    "1",
    ".",
    " **",
    "Prior",
    "itize",
    " Tasks",
    "**",
    ":",
    " Start",
    " by",
    " identifying",
    " the",
    " most",
    " important",
    " tasks",
    " that",
    " need",
    " to",
    " be",
    " completed",
    " and",
    " prior",
    "itize",
    " them",
    " based",
    " on",
    " urgency",
    " and",
    " importance",
    ".",
    " Use",
    " tools",
    " like",
    " the",
    " Eisen",
    "h",
    "ower",
    " Matrix",
    " to",
    " categor",
    "ize",
    " tasks",
    " into",
    " four",
    " quadr",
    "ants",
    ":",
    " urgent",
    " and",
    " important",
    ",",
    " important",
    " but",
    " not",
    " urgent",
    ",",
    " urgent",
    " but",
    " not",
    " important",
    ",",
    " and",
    " neither",
    " urgent",
    " nor",
    " important",
    ".",
    " Focus",
    " on",
    " completing",
    " tasks",
    " in",
    " the",
    " first",
    " quadr",
    "ant",
    " first",
    ",",
    " as",
    " they",
    " contribute",
    " the",
    " most",
    " to",
    " your",
    " goals",
    ".\n\n",
    "2",
    ".",
    " **",
    "Set",
    " Specific",
    " Goals",
    "**",
    ":",
    " Define",
    " clear",
    ",",
    " achie",
    "vable",
    " goals",
    " for",
    " each",
    " day",
    ",",
    " week",
    ",",
    " and",
    " month",
    ".",
    " Break",
    " down",
    " larger",
    " projects",
    " into",
    " smaller",
    ",",
    " manage",
    "able",
    " tasks",
    " and",
    " set",
    " deadlines",
    " for",
    " each",
    ".",
    " This",
    " helps",
    " in",
    " maintaining",
    " focus",
    " and",
    " provides",
    " a",
    " sense",
    " of",
    " accomplishment",
    " as",
    " you",
    " complete",
    " each",
    " task",
    ".\n\n",
    "3",
    ".",
    " **",
    "Use",
    " Time",
    "-",
    "Blocking",
    "**",
    ":",
    " Allocate",
    " specific",
    " blocks",
    " of",
    " time",
    " for",
    " different",
    " activities",
    " throughout",
    " your",
    " day",
    ".",
    " This",
    " involves",
    " scheduling",
    " tasks",
    " in",
    " your",
    " calendar",
    ",",
    " ensuring",
    " you",
    " ded",
    "icate",
    " time",
    " for",
    " work",
    ",",
    " breaks",
    ",",
    " and",
    " personal",
    " activities",
    ".",
    " Time",
    "-block",
    "ing",
    " helps",
    " in",
    " minim",
    "izing",
    " distra",
    "ctions",
    " and",
    " ensures",
    " that",
    " you",
    " have",
    " a",
    " structured",
    " approach",
    " to",
    " your",
    " day",
    ",",
    " maxim",
    "izing",
    " productivity",
    " and",
    " reducing",
    " pro",
    "cr",
    "ast",
    "ination",
    ".\n\n",
    "Remember",
    ",",
    " effective",
    " time",
    " management",
    " is",
    " about",
    " balancing",
    " work",
    " with",
    " rest",
    " and",
    " leisure",
    ",",
    " so",
    " ensure",
    " you",
    " schedule",
    " time",
    " for",
    " relaxation",
    " and",
    " h",
    "obbies",
    " to",
    " maintain",
    " a",
    " healthy",
    " work",
    "-life",
    " balance",
    ".",
    ""
  ],
  "started_at": "2024-07-04T01:14:12.885298Z",
  "status": "succeeded",
  "urls": {
    "stream": "https://streaming-api.svc.us.c.replicate.net/v1/streams/ilhlpztibnz5dqxtqatu33vxp5sdkfmuqobxrdcezivkrmrvtoyq",
    "get": "https://api.replicate.com/v1/predictions/hjke4jdzehrgj0cgfd684mexxg",
    "cancel": "https://api.replicate.com/v1/predictions/hjke4jdzehrgj0cgfd684mexxg/cancel"
  },
  "version": "e943ac18ac8c05ac7deb01cbc3aba827b74f1ecdcb71d722e692d7ed4e70c33d"
}

Generated in

6.9 seconds

Tweak it Share Report

Run time and cost

This model runs on Nvidia L40S GPU hardware. We don't yet have enough runs of this model to provide performance information.

Readme

Introduction

InternLM2.5 has open-sourced a 7 billion parameter base model and a chat model tailored for practical scenarios. The model has the following characteristics:

Outstanding reasoning capability: State-of-the-art performance on Math reasoning, surpassing models like Llama3 and Gemma2-9B.
1M Context window: Nearly perfect at finding needles in the haystack with 1M-long context, with leading performance on long-context tasks like LongBench. Try it with LMDeploy for 1M-context inference.
Stronger tool use: InternLM2.5 supports gathering information from more than 100 web pages, corresponding implementation will be released in Lagent soon. InternLM2.5 has better tool utilization-related capabilities in instruction following, tool selection and reflection. See examples.

InternLM2.5-7B-Chat

Performance Evaluation

We conducted a comprehensive evaluation of InternLM using the open-source evaluation tool OpenCompass. The evaluation covered five dimensions of capabilities: disciplinary competence, language competence, knowledge competence, inference competence, and comprehension competence. Here are some of the evaluation results, and you can visit the OpenCompass leaderboard for more evaluation results.

Dataset\Models	Qwen2-7B-Instruct	Yi-1.5-9B-Chat	GLM-4-9B-Chat	Llama-3-8B-Instruct	Gemma2-9B-IT	InternLM2.5-7B-Chat	Llama-3-70B-Instruct
MMLU	70.8	71.0	71.4	68.4	70.9	72.8	80.5
CMMLU	80.9	74.5	74.5	53.3	60.3	78.0	70.1
BBH	65	69.6	69.6	65.4	68.2	71.6	80.5
MATH	48.6	51.1	51.1	27.9	46.9	60.7	47.1
GSM8K	82.9	80.1	85.3	72.9	88.9	86.0	92.8
GPQA	38.4	37.9	36.9	26.3	33.8	38.4	38.9

The evaluation results were obtained from OpenCompass (some data marked with *, which means come from the original papers), and evaluation configuration can be found in the configuration files provided by OpenCompass.
The evaluation data may have numerical differences due to the version iteration of OpenCompass, so please refer to the latest evaluation results of OpenCompass.

Limitations: Although we have made efforts to ensure the safety of the model during the training process and to encourage the model to generate text that complies with ethical and legal requirements, the model may still produce unexpected outputs due to its size and probabilistic generation paradigm. For example, the generated responses may contain biases, discrimination, or other harmful content. Please do not propagate such content. We are not responsible for any consequences resulting from the dissemination of harmful information.

Open Source License

The code is licensed under Apache-2.0, while model weights are fully open for academic research and also allow free commercial usage. To apply for a commercial license, please fill in the application form (English)/申请表（中文）. For other questions or collaborations, please contact internlm@pjlab.org.cn.

Citation

@misc{cai2024internlm2,
      title={InternLM2 Technical Report},
      author={Zheng Cai and Maosong Cao and Haojiong Chen and Kai Chen and Keyu Chen and Xin Chen and Xun Chen and Zehui Chen and Zhi Chen and Pei Chu and Xiaoyi Dong and Haodong Duan and Qi Fan and Zhaoye Fei and Yang Gao and Jiaye Ge and Chenya Gu and Yuzhe Gu and Tao Gui and Aijia Guo and Qipeng Guo and Conghui He and Yingfan Hu and Ting Huang and Tao Jiang and Penglong Jiao and Zhenjiang Jin and Zhikai Lei and Jiaxing Li and Jingwen Li and Linyang Li and Shuaibin Li and Wei Li and Yining Li and Hongwei Liu and Jiangning Liu and Jiawei Hong and Kaiwen Liu and Kuikun Liu and Xiaoran Liu and Chengqi Lv and Haijun Lv and Kai Lv and Li Ma and Runyuan Ma and Zerun Ma and Wenchang Ning and Linke Ouyang and Jiantao Qiu and Yuan Qu and Fukai Shang and Yunfan Shao and Demin Song and Zifan Song and Zhihao Sui and Peng Sun and Yu Sun and Huanze Tang and Bin Wang and Guoteng Wang and Jiaqi Wang and Jiayu Wang and Rui Wang and Yudong Wang and Ziyi Wang and Xingjian Wei and Qizhen Weng and Fan Wu and Yingtong Xiong and Chao Xu and Ruiliang Xu and Hang Yan and Yirong Yan and Xiaogui Yang and Haochen Ye and Huaiyuan Ying and Jia Yu and Jing Yu and Yuhang Zang and Chuyu Zhang and Li Zhang and Pan Zhang and Peng Zhang and Ruijie Zhang and Shuo Zhang and Songyang Zhang and Wenjian Zhang and Wenwei Zhang and Xingcheng Zhang and Xinyue Zhang and Hui Zhao and Qian Zhao and Xiaomeng Zhao and Fengzhe Zhou and Zaida Zhou and Jingming Zhuo and Yicheng Zou and Xipeng Qiu and Yu Qiao and Dahua Lin},
      year={2024},
      eprint={2403.17297},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

vLLM Deployment

This model was created by the Replicate Model: Create a new vLLM model