What is CrewAI, and why do I care?

Mar 7

Why would I need/want this?

CrewAI is a platform that enables more complex workflows to be automated using AI to simulate roles. If I wanted to use ChatGPT to write an email for me, and that was it, then CrewAI would be overkill. However, if I then wanted to take that email and make a LinkedIn post, do some research, or perform any other tasks using that information, ChatGPT can often become confused and stop using the information it provided initially. This is where CrewAI steps in.

You can define agents that act independently or together to finish tasks. For instance, suppose you wanted to create job postings; what roles or tasks would you define? You may use a research analyst to generate the description and incorporate company culture, values, and specific needs beyond job tasks. Then, you may hand that research to the writer, who will take that information and formulate the job posting. Then, you have to review it and make sure that it is clear, engaging, and aligned with company values and HR requirements (state/local/federal laws, etc.).

This concept is especially useful for complex processes repeated regularly (e.g., job posting writing). Does this intrigue you? Do you want to try it out? Below are the steps to get you started; they come from the project documentation.

Step 0: Installation

Installation is straightforward; you need the following Python packages installed:

crewai
crewai[tools]
duckduckgo-search # or whatever other tools apply to your project

Step 1: Assemble your agents

Your agents need to be defined, you give them a name, role, and backstory. For example, in the code below, the `Agent` agent is given the role of `Senior Researcher`. The AI is then given a goal, and a backstory. This helps the AI define what each agent does, what tools it has access to, wether its allowed to delegate (i.e., task other agents to perform some work).

import os
os.environ["OPENAI_API_KEY"] = "Your Key"

from crewai import Agent
from langchain_community.tools import DuckDuckGoSearchRun
search_tool = DuckDuckGoSearchRun()

# Topic for the crew run
topic = 'AI in healthcare'

# Creating a senior researcher agent with memory and verbose mode
researcher = Agent(
  role='Senior Researcher',
  goal=f'Uncover groundbreaking technologies in {topic}',
  verbose=True,
  memory=True,
  backstory="""Driven by curiosity, you're at the forefront of
  innovation, eager to explore and share knowledge that could change
  the world.""",
  tools=[search_tool],
  allow_delegation=True
)

# Creating a writer agent with custom tools and delegation capability
writer = Agent(
  role='Writer',
  goal=f'Narrate compelling tech stories about {topic}',
  verbose=True,
  memory=True,
  backstory="""With a flair for simplifying complex topics, you craft
  engaging narratives that captivate and educate, bringing new
  discoveries to light in an accessible manner.""",
  tools=[search_tool],
  allow_delegation=False
)

Step 2: Define your tasks

Here we will define what tasks we want our agents to perform. Such as summarizing articles, researching trends, writing code, etc. Below is a few examples:

from crewai import Task

# Research task
research_task = Task(
  description=f"""Identify the next big trend in {topic}.
  Focus on identifying pros and cons and the overall narrative.
  Your final report should clearly articulate the key points,
  its market opportunities, and potential risks.""",
  expected_output='A comprehensive 3 paragraphs long report on the latest AI trends.',
  tools=[search_tool],
  agent=researcher,
)

# Writing task with language model configuration
write_task = Task(
  description=f"""Compose an insightful article on {topic}.
  Focus on the latest trends and how it's impacting the industry.
  This article should be easy to understand, engaging, and positive.""",
  expected_output=f'A 4 paragraph article on {topic} advancements fromated as markdown.',
  tools=[search_tool],
  agent=writer,
  async_execution=False,
  output_file='new-blog-post.md'  # Example of output customization
)

In the example above we have a research task and a writing task. Notice that the tasks are broken down into verbs (i.e., research, and write, rather than research and write). separating them out overall leads to better outcomes. And that the task is assigned to an agent, this way the agent knows what tasks it should perform. Additionally, because they are broken out, they can be performed asynchronously. For example, as one agent is writing, the researcher can begin researching the next topic.

Step 3: Build the crew

In this step, we define the mapping between agents and tasks.

from crewai import Crew, Process

# Forming the tech-focused crew with enhanced configurations
crew = Crew(
  agents=[researcher, writer],
  tasks=[research_task, write_task],
  process=Process.sequential  # Optional: Sequential task execution is default
)

Here, we set up the tasks and agents and specify the type of process execution (sequentially or hierarchal). If you choose hierarchal, you must specify a manager to coordinate the workflow. The manager will delegate tasks, validate outcomes, and ensure execution.

Here is an example of a hierarchal setup:

from langchain_openai import ChatOpenAI
from crewai import Crew, Process, Agent

# Agents are defined without specifying a manager explicitly
researcher = Agent(
    role='Researcher',
    goal='Conduct in-depth analysis',
    # tools = [...]
)
writer = Agent(
    role='Writer',
    goal='Create engaging content',
    # tools = [...]
)

# Establishing the crew with a hierarchical process
project_crew = Crew(
    tasks=[...],  # Tasks to be delegated and executed under the manager's supervision
    agents=[researcher, writer],
    manager_llm=ChatOpenAI(temperature=0, model="gpt-4"),  # Defines the manager's decision-making engine
    process=Process.hierarchical  # Specifies the hierarchical management approach
)

Notice that tools in the agent definitions are commented out, meaning, the manager handles what tools are executed by which agent based on the role and goal. This allows a more complex workflow to be a bit more "fire and forget".

Step 4: Kick off the workflow

In this step, we initiate the workflow, we can watch as agents collaborate, and see our project come to life.

# Starting the task execution process with enhanced feedback
result = crew.kickoff()
print(result)

Step 5: Crack open a cold beverage; you have earned it!

This is just an example of a workflow, but it should give you sufficient insight to see how it could be useful for processes where an AI could help. However, because of an AI's chat nature, it’s not conducive.

Sean Fay