Manage Code Dependencies with Symlinks
A sample demonstrating how to reference common files and folders using symbolic links, with limitations noted and an alternative approach recommended.
Why it matters
This asset demonstrates how to effectively use symbolic links (symlinks) within Promptflow to manage and reference common files or folders across different parts of your project. It's designed to simplify dependency management in complex code structures.
Outcomes
What it gets done
Implement symlink-based file referencing in Promptflow.
Structure project directories for efficient resource sharing.
Troubleshoot common issues related to symlink usage in code flows.
Integrate with version control systems like GitHub for symlink management.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pf-standard-flow-with-symlinks | bash Capabilities
What this chain does
Writes source code or scripts from a description.
Traces errors to their root cause and suggests fixes.
Runs build pipelines, tests, and deploys to environments.
Overview
Flow With Symlinks
What it does
A sample showing symlink usage for referencing common files/folders, with noted limitations and a recommended alternative.
How it connects
The source recommends using additional includes instead due to symlink limitations.
Source README
Flow with symlinks
User sometimes need to reference some common files or folders, this sample demos how to solve the problem using symlinks.
But it has the following limitations. It is recommended to use additional include.
Learn more: flow-with-additional-includes
- For Windows user, by default need Administrator role to create symlinks.
- For Windows user, directly copy the folder with symlinks, it will deep copy the contents to the location.
- Need to update the git config to support symlinks.
Notes:
- For Windows user, please grant user permission to create symbolic links without administrator role.
- Open your
Local Security Policy - Find
Local Policies->User Rights Assignment->Create symbolic links - Add you user name to this policy then reboot the compute.
- Open your
Attention:
- For git operations, need to set:
git config core.symlinks true
Tools used in this flow
- LLM Tool
- Python Tool
What you will learn
In this flow, you will learn
- how to use symlinks in the flow
Prerequisites
Install promptflow sdk and other dependencies:
pip install -r requirements.txt
Getting Started
1. Create symbolic links in the flow
python ./create_symlinks.py
2. Test & run the flow with symlinks
In this sample, this flow will references some files in the web-classification flow, and assume you already have required connection setup.
You can execute this flow or submit it to cloud.
Test flow with single line data
# test flow with default input value in flow.dag.yaml
pf flow test --flow .
# test flow with input
pf flow test --flow . --inputs url=https://www.youtube.com/watch?v=o5ZQyXaAv1g answer=Channel evidence=Url
# test node in the flow
pf flow test --flow . --node convert_to_dict --inputs classify_with_llm.output='{"category": "App", "evidence": "URL"}'
Run with multi-line data
# create run using command line args
pf run create --flow . --data ./data.jsonl --column-mapping url='${data.url}' --stream
# create run using yaml file
pf run create --file run.yml --stream
You can also skip providing column-mapping if provided data has same column name as the flow.
Reference here for default behavior when column-mapping not provided in CLI.
Submit run to cloud
# create run
pfazure run create --flow . --data ./data.jsonl --column-mapping url='${data.url}' --stream --subscription <your_subscription_id> -g <your_resource_group_name> -w <your_workspace_name>
# set default workspace
az account set -s <your_subscription_id>
az configure --defaults group=<your_resource_group_name> workspace=<your_workspace_name>
pfazure run create --file run.yml --stream
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.