News

Copilot Rolled Out to Your Org? Better Check User Activity

• By David Ramel
• 08/12/2024

Many organizations jumped on the Gen AI bandwagon in a big way, immediately appointing AI execs and rolling out AI assistants to many -- if not all -- employees. For us here at Visual Studio Magazine, that means developers, who probably were granted access to GitHub Copilot, the original "AI pair programmer" that kicked off Microsoft's "AI era" years ago.

But now that advanced AI has fallen into the "trough of disillusionment" stage of the hype cycle, it might be time to check user activity for Copilot in your organization, for both developers and other staffers who use one of Microsoft's seemingly uncountable Copilots across a wide swathe of products and services.

Reviewing this data can help an organization optimize its investment in Copilot, ensuring it supports their development processes effectively. It can also help find and mitigate inappropriate or risky behavior. Potential benefits include:

• Resource Allocation: Organizations can ensure Copilot licenses are efficiently used by monitoring active usage, reallocating seats as needed to maximize productivity.
• Cost Management: By tracking usage, organizations can manage subscription costs, potentially downgrading or upgrading based on actual needs.
• Security and Compliance: Monitoring activity helps in ensuring that Copilot usage aligns with organizational policies and security standards.

Luckily, recent documentation from both Microsoft and Microsoft-owned GitHub provides guidance on how to review user activity data for Copilot in an organization.

For example, a few weeks ago, Microsoft published "Microsoft 365 reports in the Admin Center -- Copilot for Microsoft 365 usage."

It states: "The Microsoft 365 Usage dashboard shows you the activity overview across the Microsoft 365 productivity apps in your organization. It enables you to drill into individual product-level reports to give you more granular insight about the activities within each app."

Managers can view a summary of users' adoption, retention, and engagement with Copilot for Microsoft 365, and the activity of every Copilot user in the organization, with reports generally available within 72 hours of the end of a day's activity.

Speaking of dashboards, Microsoft earlier this year announced "Microsoft Copilot Dashboard now generally available."

The announcement said: "Today, we are pleased to announce the general availability of the Microsoft Copilot Dashboard powered by Viva Insights, a new and improved way to understand the impact of your Copilot for Microsoft 365 investment. Originally announced at Microsoft Ignite 2023, the Copilot Dashboard provides privacy protected insights across every stage of the transformation journey, and offers organizations the ability to understand the impact of Copilot on common workplace activities such as meetings, email, chat, documents, and search."

Just last week, Microsoft published associated guidance titled "Connect to the Microsoft Copilot Dashboard for Microsoft 365 customers."

It states: "The Microsoft Copilot Dashboard in Viva Insights helps organizations maximize the value of Copilot for Microsoft 365. It provides actionable insights to help your organization get ready to deploy AI, drive adoption based on how AI is transforming workplace behavior, and measure the impact of Copilot."

Other Microsoft guidance includes "Analyze copilot performance and usage in Copilot Studio," which provides summary charts and data on engagement over time, session outcomes over time, resolution rate drivers and more.

For example, it states: "The Summary tab of the Analytics page gives you a broad overview of your copilot's performance. It uses artificial intelligence (AI) technology to show you which topics are having the greatest impact on escalation rate, abandon rate, and resolution rate."

'Inappropriate or Risky Interactions'
In addition to checking usage data to control costs, efficiently allocate resources and so on, Microsoft Purview guidance includes, "Configure a communication compliance policy to detect for Copilot for Microsoft 365 interactions."

It states: "You can use communication compliance to analyze interactions (prompts and responses) entered into Copilot for Microsoft 365 to detect for inappropriate or risky interactions or sharing of confidential information."

It works across a variety of Copilot apps:

• Excel Copilot
• Forms Copilot
• Loop Copilot
• Microsoft 365 Chat in Bing
• Microsoft 365 Chat in Teams
• OneNote Copilot
• Outlook Copilot
• Planner Copilot
• PowerPoint Copilot
• Stream Copilot
• Teams (chats/channels/meetings) Copilot
• Word Copilot
• Whiteboard Copilot

Interactions covered include:

• Sensitive information types
• Keywords
• Trainable classifiers
• Content safety classifiers for Teams based on large language models
• Conditions

GitHub Weighs In
GitHub, meanwhile, published "Reviewing user activity data for Copilot in your organization" in order to help "Review GitHub Copilot usage in your organization to make informed decisions about seat assignment."

That can help in cost management.

GitHub also published "Viewing Copilot license usage in your enterprise" in order to "View how many users have access to GitHub Copilot across all the organizations in your enterprise."

It states: "If you have a subscription to GitHub Copilot Business or GitHub Copilot Enterprise, you can view usage information for GitHub Copilot in your enterprise, broken down by organization, or in your organization, broken down by seat assignment status. At the enterprise level, this information includes the number of seats assigned in each organization, and the total spending associated with each organization, for the current billing cycle. At the organization level, this information includes the total number of seats, seats carried over from the previous billing cycle, new seats added during the current cycle, and seats to be removed at the end of the current cycle."

Microsoft Copilot Weighs In
We asked Microsoft Copilot why an organization would want to track Copilot usage activity. It replied:

Organizations track Copilot usage activity for several key reasons:

• Resource Optimization
  By monitoring how Copilot is used, organizations can ensure that the tool is being utilized effectively. This helps in reallocating resources to those who benefit the most from it, ensuring that the investment in Copilot is justified and maximized.
• Cost Management
  Tracking usage helps in identifying inactive or underutilized licenses. This allows organizations to manage their subscriptions more efficiently, potentially reducing costs by reallocating or canceling unnecessary licenses.
• Productivity Insights
  Understanding how Copilot impacts productivity can provide valuable insights. Organizations can assess whether the tool is helping developers code more efficiently and identify areas where additional training or support might be needed to enhance its effectiveness.
• Security and Compliance
  Monitoring user activity can help in detecting any unusual or unauthorized usage patterns, ensuring that the organization's security and compliance standards are maintained. This is particularly important for protecting sensitive code and data.
• User Engagement and Support
  By tracking which team members are actively using Copilot, organizations can identify those who might need additional support or training. This ensures that all team members can fully leverage the tool's capabilities, leading to better overall performance.
• Informed Decision-Making
  Access to detailed usage data allows organizations to make data-driven decisions regarding tool adoption and usage policies. This can lead to better strategic planning and more effective implementation of development tools.