IND US 20190354912A1 IN ( 19) United States (12 ) VARMA PatentetApplication al. Publication ((1043)) Pub Pub..No .: US 2019/0354912 Date : A1 Nov. 21, 2019 (54 ) SYSTEM AND METHOD FOR SOFTWARE Publication Classification ADOPTION (51) Int. Ci. ( 71) Applicant: APPLANGO SYSTEMS LTD ., Rosh G06Q 10/06 ( 2006.01) Ha'ayin ( IL ) (52) U.S. CI. CPC . G06Q 10/06375 (2013.01); G06Q 10/06393 ( 72 ) Inventors: Samir VARMA, Cos Cob , CT (US ); (2013.01) Daniel SARFATI , Raanana (IL ) (73) Assignee : APPLANGO SYSTEMS LTD ., Rosh ( 57 ) ABSTRACT Ha'ayin ( IL ) (21) Appl. No.: 16 / 349,769 Systems and methods are described for determining one or more benchmarks for software usage of an organization ( 22 ) PCT Filed : Nov. 14 , 2017 based on data internal to the organization . In one example , usage data of a software application of an organization may ( 86 ) PCT No .: PCT /US17 /61641 be obtained or collected . One or more expected users of the software application may be identified . Next, at least one $ 371 (c ) (1 ), benchmark for usage of the software application may be ( 2 ) Date: May 14 , 2019 determined based on a power law , the identified expected Related U.S. Application Data user or users, and usage data internal to the organization . The obtained usage data may then be compared to the one (60 ) Provisional application No.62/421,924 , filed on Nov. or more benchmarks, and results of the comparison output, 14 , 2016 . for example via a graphical user interface or a report. Software Adoption Health Check - 2002 Number of Monitored Users : 572 (SFDC ) Date : 1. Overall Adoption Health Status Login Frequency 205 Login Frequency / Company Average Customer * Infrequent users, Infrequent users 23 % 37 % 5% 11 % 30 % 51% Login Frequency / Healthy 38 % 59 % 25 % 79 % Sep - 15 Oct- 15 > monthly Monthly Bi-weekly Weekly * Average Customer: calculated on first month of monitoring Patent Application Publication Nov. 21, 2019 Sheet 1 of 13 US 2019/0354912 A1 -100 105 obtain access information and usage data Yes 110 Determine if No active user No 120 generate indication to 150 Identify expected user by Verify user needs remove functionality of application application license Yes 155 Determine benchmark (per functionality ) for comparing actual 130 generate usage data 125 No indication to verify user has valid deliver valid credentials credentials 160 Present options to configure | analysis/asusmptions Yes 135 140 generate 165 Compare benchmark (s ) with verify user has No indication to actual usage data adequate provide training training 170 Output results of comparison 145 add user to active user list 115 Trigger indication if results below benchmark Fig. 1 Patent Application Publication Nov. 21, 2019 Sheet 2 of 13 US 2019/0354912 A1 20 3 37 % 10 % HFLreoaql/ugetinhcyn % 5 10 % CAuvset*romaegre % 23 % % 30 % : Date 25 % cCAmfaouvlnsiectur*:nfloarmstigenhrdg 11 % ASHCdoehfaptewlicatorkhne 205 Inufsre,qurenst % Oct 15 wWeeOBi-kly 2A . Fig 59 % SFDC ) ( FCLroemqg/upeaincny Monthly % % MNU5ounsimf7:etbore2rsd AOHSdvetoap1.rtlaitulolhsn Inufrseq,urenst 59 15 Sep mont>hly FLreoqgueincny % 51 Patent Application Publication Nov. 21 , 2019 Sheet 3 of 13 US 2019/0354912 A1 20 5 % 9 139 FHLreoaql/ugetinhcny 6 % CAuvset*romaegre 36 % 11 % 72 % cmCAfaouvlnisceturn:f.loarmstigenhrdg aNctoivne 308,users mumu UU K w WweeBi-kly 15 Oct 2B . Fig 29 % AUCcostmiapvagitney DU aNctoivne 266,users mMon0>thly 14 % 11 % 15 Sep AUcstiavigtey % 47 Patent Application Publication Nov. 21, 2019 Sheet 4 of 13 US 2019/0354912 A1 300a 310 OIHmpetraoi-lvemtahenlyt60-10 OHImpetaroi-lvmetmahelny6t0-100 Ne d 40-60 320 Ne d 40-60 Sacrament Date : X AUcrtgieontRequired bunul AUcrtgieontRequired E 305 R a n g e 572 100 Scale ) ( Range 100 100 100 100 0 0 SAHCodehfatpewlcaitrokhen Reference 40 572 Reference 40 40 3A . Fig ( Good Min150 )iOct Sepmal 12 61 715 ( SFDC ) Results 57 168300 Results 1500-SepOct 27 60 30 61 23 63 24 UAcst63%eirvse MNU5ounsime:f7tbore2srd AUcvtsiet408vperitaviges OUApcsotr%108eiurnvistey UAcosnt81e%iarvcste OCTrgareni2.oszatiosn FALrcetqivueintclyNUFusmofeblrs CLoleabovraetioln AOdbojp3.teicotnoubsjTopeyacgtesocrgranio--zastion FRreaqutencey FRreaqutencey FRreaqutencey Item Adoption Colaboration Colaboration Colaboration Lead Patent Application Publication Nov. 21, 2019 Sheet 5 of 13 US 2019/0354912 A1 30 0 OHImpetaroi-lvmetmahelny6t0-10 JAN Ne d 40-60 AUcrtgieontRequired1-40 315 Range 100 100 0 100 0 100 100 100 3Av2e..ctr5ag6e 40 1Av1e6.ctr.ag3e 40 40 2Av7e9.ctr.ag4e 40 40 3Av2e6.ctr.ag2e 7Av4e1.ctr.ag9e 40 40 3B . Fig Reference hd 57 58 38 59 55 50 Results AUcst36%eirvse 150 Oct Sep 53 58 AUcste53%irvse 34 53 45 60 AUcst46%eirvse 51 AUcste%70irvse 42 54 ASTUcarsterProd-LS70%lainvse Gurso5Topbyaugse User--C&ASales FRreaqutenecy SalUser-LSs FRreaqutenecy FRreaqutencey MSaUrksDetiLnrg FRreaqutenecy Operations FRreaqutenecy SGtrao4.tups Collaboration Colaboration Colaboration Colaboration Colaboration Patent Application Publication Nov. 21, 2019 Sheet 6 of 13 US 2019/0354912 A1 40 a Company 25 % OHImpetaroi-lvmetmahelnyt % 29 29 % 31 % 15 Oct Ne d 40-60 % 33 20 % % 16 Date : UArcgteinotRequired U A cst i a v i g tey 26 % % 32 Sep 15 - & C A Sales User 30 % 12 % Range 100 O 100 0 ROenaldy UsLowage FUuslage 4A SAHCodehfoatpewlcaitrokhen pOsigefpvnretocerfimuaeplnisrcet Reference 40 11 % No 5 % Login 25 $ % Company Fig . 59 % 50 SFDC ) ( Results AUvcset36%33.trivasege conta-15Octi1502Sepning 5758 6 % 13 % 44 % % 37 Oct FLreoqgueincny 14 % % 40 Sales A & C User 15 Sep MNUo5unsimte7f:bore2srd % GroInZoom5.ups UserSalesC&A--FCollaboration requency mon>thly Monthly We kly weBi -kly Patent Application Publication Nov. 21, 2019 Sheet 7 of 13 US 2019/0354912 A1 40 5 29 % Company % OIHmpetraoi-lvmetahelny6t0-100 29 % 17 % KAN Do 9 % 43 % Oct 15 Ne d 40-60 22 % 48 % AUcrtgieontRequired1-40 A U cst i a vgi t e y 11 % Sep 15 RUensewealrs 56 % Range 100 100 o ReadOnly UsLowage UsFullage 11 % No Login 4B . Fig Reference 25 % nanah 40 40 5 % % 59 Company Results AUvcste3336%.tirvasege 150-OctSep 6560 12 % 88 % Oct FLreoqgueincny @ RUensewealrs 111 Sep 15 89 % FColaboration requency mon>thly Monthly DWe kly User Sales A & C weBi -kly Patent Application Publication Nov. 21, 2019 Sheet 8 of 13 US 2019/0354912 A1 40 C Company 25 % 6OIHmpetraoi-lvmetahelnyt0-10 % 29 29 % % 0 % 9 % Oct 15 Ne d 40-60 % 82 InsSales Useride A & C AUcrtgieontRequired1-40 U A cst i a vgi t e y 6 % 7 % % 15 Sep 81 % Range 0 LoNogin ROenaldy UsLowage FUusalge 4C . Fig Reference % Company 11 % 5 % Ave33.ctrage 59 % Results Sep 6551 S659 150 Oct - % 91 % Oct 15 InsUser C & A Saleside FLreoqgueincny 0 % % 15 100 Sep 36 Users Act% & C Sales Userive m o n t h ly FColaboration requency Monthly weeNBi-kly We kly Patent Application Publication Nov. 21 , 2019 Sheet 9 of 13 US 2019/0354912 A1 505 obtain access information and usage data for group $ 10 Determine benchmark ( per functionality ) for comparing actual usage data 515 Compare benchmark (s ) with actual usage data 520 Obtain performance information of group 525 Correlate usage data with performance information 530 determine scale of usage data based on benchmark ( s ) and correlated performance information 535 Display usage data relative to scale Fig . 5 Patent Application Publication Nov. 21, 2019 Sheet 10 of 13 US 2019/0354912 A1 AN SFDC|0001a --- Franchise Rep User wPardot and Engage ////? ////// . ! ZIUM Fig . 6A Account Contact Event Lead Log-in Opportunity Task 1.432 1.200 1,000 600 200 o NION . 2016-12-03 2016-12-09 2016-12-15 2016-12-21 2016-12-27 Fig . 6B Patent Application Publication Nov. 21, 2019 Sheet 11 of 13 US 2019/0354912 A1 Handover Frequency Rate Collaboration Rate SFCC /Catania / Alto Fig . 7A SFDC /Central Alto Total : 783 Handovers : 62 % No Handovers Handovers 2016-1 -20 2016-11-23 2016-1 -26016-1 -292016-12-20216-12-025016-12-08 2016-12-21016-12-14 2016-12-17 Fig . 7B Patent Application Publication Nov.. 21, 2019 Sheet 12 of 13 US 2019/0354912 A1 90 80 70 UR 60 50 80 % 2 Opportunity 40 Lead 30 N Contact Page 22-Dec-1429-Dec-1405-Jan-1512-Jan-1519-Jan-1526-Jan-1502-Feb-1509-Feb-15 24-Nov-1401-Dec-1408-Dec-1415-Dec-14 Fig . 8A 30 20 Opportunity a Lead O Contact 5 M Sep-1409 03- Sep -14* 16-Sep -1423-Sep-143007-001-14 Sep-14 14-Oct-1421-Oct-1428-Oct-1404-Nov-1411-Nov-1418-Nov-14 Fig . 8B Patent Application Publication Nov. 21, 2019 Sheet 13 of| 13 US 2019/0354912 A1 8 Opportunity 2 Lead » Contact -1413 06-Jul -1420-Jul-1427-Jul-14 -Jul 03Aug 24-Aug1431-Aug-1407 -1410-Aug-1417-Aug-14 --1414Sep 21 Sep-14-Sep-14 Fig. 8C 6 ALLAH 2 Opportunity E Lead termind S Contact 0 I 24-Nov-1401-Dec-1408-Dec-1415-Dec-1422-Dec-1429-Dec-1405-Jan-1512-Jan-1519-Jan-1526-Jan-1502-Feb-15 LI-AON-I+ Fig . 8D US 2019/0354912 A1 Nov. 21, 2019 SYSTEM AND METHOD FOR SOFTWARE [0007 ] FIG . 5 illustrates an example flow diagram of a ADOPTION process for analyzing and categorizing software usage within a group of users. COPYRIGHT NOTICE [0008 ] FIGS . 6A , 6B , 7A , 7B , and 8A - 8D illustrate [0001] This disclosure is protected under United States example analyzes of software usage . and/ or International Copyright Laws. © 2016 Applango Sys DETAILED DESCRIPTION OF THE tems, Ltd. A portion of the disclosure of this patent docu PREFERRED EMBODIMENT ment contains material which is subject to copyright pro tection . The copyright owner has no objection to the Software Adoption facsimile reproduction by anyone of the patent document or the patent disclosure , as it appears in the Patent and /or [0009 ] The concept of software adoption is easily tied to Trademark Office patent file or records, but otherwise the number of people who have access to software . Tracking reserves all copyright rights whatsoever. user logins is a basic indicator that provides clear and initial BACKGROUND proof of usage, access , and/or activity. Software as a service applications (SaaS) as well as single sign -on applications [0002 ] When an organization desires to launch an enter (SSO ) often provide reports on login activities . prise -wide software application , the challenge often lies in [0010] For mission critical applications ( i.e., those appli how to identify the proper software application to purchase , cations that are critical to the daily operations and success of how to implement the chosen software application , and how the business/organization ), however, proof of login is not to measure the usage activity of the software application . For enough to provide a good indicator of software adoption of example, an organization often buys licenses based on what usage . In addition to access, organizations need to under the salesmen convince them to purchase , rather than buying stand in fine detail how the applications are being used . A licenses which the organization needs. Similarly, people complete list of every activity performed by every user,with leave and new people join the organization , so it can also be its timestamp, can be obtained through log files and /or difficult to ensure there is the proper training in place so that through probes installed on the software. This is known as people know “ how , when , and why ” they should use the usage data . software application . When trying to understand the value [0011 ] In order to measure the degree of software adoption and effectiveness of the software application , the organiza via usage data, usage data should be compared to a bench tion may send polls, but people will often lie on polls and /or mark or standard . For example , an optimal usage data respond with the answers they believe the organization pattern is likely one that is in line with management direc wishes to see . tives and expectations on which business operations/process [0003] Moreover , even when there is data collected , it can should be performed on such applications and how the use be difficult to make any meaningful determinations. More of the software applications are intended to meet the busi usage does not necessarily reflect a more successful adop ness objectives. This involves a comparison based on tion as employees are not expected to spend 100 % of their expected usage, by activity and by user , at a very high level working time on any particular application . There is also no of detail. However, even the managers themselves may not standard set for accepted levels of software usage, and be able to produce a detailed and quantifiable list of expected external data points are not generally available for compari activities to be performed and by whom . Much of this son . information is outside the scope of the normal managerial attention level. SUMMARY OF THE INVENTION [ 0004 ] One of the metrics utilized to determine the success Implied Comparison Values and effectiveness of a software application is to measure its [0012] The present disclosure addresses this problem by adoption rate . The methodology disclosed in this applica generating one ormore implied comparison values for usage tion, in one aspect, creates an objective benchmark to data via an algorithm , with or without knowing manager measure the “ health ” of user software adoption of a software expectations . application within an organization . In another aspect, the [0013] FIG . 1 illustrates an example process 100 for benchmark may be utilized to identify abnormal software determining one or more usage benchmarks. It should be usage to direct positive change in a group of users of the appreciated that operations indicated by dashed lines are software. In yet one aspect , the usage of software by a group optional. It should also be appreciated that in some imple may be analyzed to determine normal, sub -normal, and mentations, one or more operations may be omitted from exceptional usage patterns to inform procedural changes for process 100. Process 100 begins with obtaining access or individual users within a group of users , or across an login information from or of a software system under organization. analysis , at operation 105. Next, that information may be BRIEF DESCRIPTION OF THE DRAWINGS divided into two groups of users (those with access to a license ): those with proof of login (" active users” ) and those [ 0005 ] FIG . 1 illustrates an example flow diagram of a who have never logged in (" inactive users” ), at operation process for determining Overall Adoption Health Status and 110. The reason for the divide is that if there is overall Usage Activity of a Company . low -adoption of the software, there is a difference between [0006 ] FIGS. 2A , 2B , 3A , 3B , 4A , 4B , and 4C illustrate an low - adoption users who know how to login and users who example graphical user interface displaying charts showing never logged in , and thus should be categorized differently. the Overall Adoption Health Status and Usage Activity of a [0014 ] For the first group of active users , the objective is Company. to identify why adoption is low and motivate users to US 2019/0354912 A1 Nov. 21, 2019 interact more with the application . In the second group of [0020 ] In some cases, a benchmark may be determined at inactive users , the objective becomes verifying that they operation 155 for usage of an entire application , program , need the application operation 115 ), that they have valid etc. , and not be broken into functionality or object. credentials (operation 125 ), and that they are trained in using the application (operation 135 ). Various actions can be Benchmarking performed in response to a determination that any of need , [0021] Benchmarking is common in business and its use credentials , or training is lacking (at operations 120, 130 , fulness in contextualizing performance is widely accepted . 140 ). In the event that the inactive user needs the applica As discussed , creating a benchmark usually requires data tion , has valid credentials, and is properly trained to use the that is external to that which is being benchmarked . For software , the user may then be added to the active user list many business functions, such external data is commonly (operation 145 ), or some other action may be taken . available . Consider the following examples : [0015 ] Among active users, the question still remains: [0022 ] To benchmark how efficient your hospital is, you how to identify " low ” adoption , or in other words, how to would collect data from many other hospitals and compare define “ expected usage.” It is inherently difficult to ascertain yourself to them . which users should use the various functionalities of the application and the rate or frequency at which usage should [0023] The University of Chicago Law school is ranked # 4 happen , if at all. amongst all law schools in the United States . This is a [0016 ] The first part of “ expected usage " is identifying the benchmark , and the result is 4. The data is external to the “ expected user” of a particular functionality , at operation University of Chicago — that is, you need data from ALL law 150. If a user has touched a functionality, it is assumed that schools in the US . he/she is an expected user of that functionality. [0024 ] GE aero engines use 20 % less floor space in its [0017 ] The next issue is to determine what the expected factory to produce jet engines than SNECMA and Rolls pattern of activity is for an expected user, a given group of Royce. This is a benchmark , and the number is 20 % . You users, and /or for the whole company . Typically , the method need data external to GE (that is, you need data regarding the of determining expected pattern of usage would be to create floor space of engines from the other aero engine manufac a benchmark by looking at the patterns of other, similar, turers ). organizations, and comparing one's own activity to that of [0025 ] The profits per partner of Simpson Thatcher are $ x others . However, in this case : per year and they are higher than all but y law firms in NY. [0018 ] (1 ) No such data exists because the software ven This is a benchmark , the number is x , and the data is external dors do not collect it . The data contemplated is very to Simpson Thatcher (you need data from all NY law firms). granular software usage data: every activity by every user And so on . on every object . It would entail collecting a record of [0026 ] For software adoption , company - to -company com every single activity on every single object by every parisons are nearly impossible and/or worthless because the single user on the application system . This is millions external data does not exist, because no one collects it , and upon millions of pieces of data every week . For example , thus there is no benchmark . And even if the data can be imagine that 5000 users update/create and delete an collected , the precise comparable organization to benchmark average of 100 objects per day. That's 500,000 activities against may be extremely difficult or impossible to find . per day that we have recorded , which is 5x500,000 = 2.5 Fitted Distributions million activities per week . Then , that data has to be analyzed in some way to oduce useful business value . [0027 ] To address this and other problems, a way of (2 ) There is no way of knowing which external organi deriving an expected usage pattern ( i.e., benchmark ) from zations are relevant to benchmark against. Should the the data itself has been created . To determine acceptable organizations be the same size , have the same number of versus un -acceptable usage levels, an extensive research licenses , same age of installation , revenue per employee , project to identify a “ minimal usage pattern ” was conducted number of countries operated in , number of divisions, That is , a usage pattern such that if the group /user/company etc. ? is not at least hitting that pattern , there is a good chance that [0019 ] As will be described in more detail below , one or the usage is unhealthy. more benchmarks may be determined at operation 155 , for [0028] The described solution includes the manipulation example, based on usage of an entire application , or one or of two mathematical theories and combines them in order to more specific functionalities thereof. The one or more provide an answer to benchmarking using only internal data benchmarks may be determined directly from the collected and , as a bonus, provide insights for positive organizational data without need of or reference to data from another change . organization or application : in other words, a self-generating benchmark . Actual usage data may then be compared to the Gini Coefficient one or more of these self-generated benchmarks, at opera tion 160. In some cases , one or more options to configure or [0029 ] One important aspect of determining an internal modify the analysis may be presented at operation 165, for data -based benchmark is determining the level of “ equality example , to adjust the usage level used as a benchmark . The of usage” in the system . There is a well -known formula in results may then be displayed or delivered at operation 170 . economics called the “ Gini Coefficient” (GC ). The GC In some cases, one or more indications may be triggered if measures the extent of inequality in any distribution of the results fall below the one or more benchmarks, at outcomes.Most classically , the GC is a measure of statistical operation 175. Ways in which to determine the one ormore dispersion intended to represent the income distribution of a benchmarks, as performed at operation 155 , will be nation's residents, typically the inequality in wealth and in described below . income. US 2019/0354912 A1 Nov. 21, 2019 (0030) AGC of zero (0 ) expresses perfect equality: every rule is roughly followed by a power law distribution for a one has the same level of the value that is being measuring . particular set of parameters (here, known as a Pareto distri A GC of one ( 1) represents perfect inequality : for example , bution ). only one person has all the income or wealth and the others [0035 ] In particular , the distribution of wealth follows a all have zero . Pareto distribution (80 % of the wealth in Italy was owned by [ 0031 ] For the present solution , 1 - GC is an excellent 20 % of the people ), and the Italian economist that this indicator of usage health : if there was zero inequality, then principle is named after, Vilfredo Pareto , developed these 1 -GC = 1-0= 1 = 100 % . This would mean everyone is using concepts in the context of the distribution of income and the application equally. While measuring usage equality was wealth among the population . It is also a common rule of an objective , however, 100 % as the benchmark is not a good thumb in business: e.g., “ 80 % of your sales come from 20 % indication of ideal or optimized usage. 100 % as a benchmark of your clients.” In software engineering, Lowell Arthur would fail because it is completely impossible for every user expressed a corollary principle: “ 20 percentof the code has interacting with the application to interact with it exactly the 80 percent of the errors.” Many natural phenomena have same amount. Furthermore, who's to say that this should be been shown empirically to exhibit such a distribution : 80 % the target companies are striving for. Some people will of the peas in Vilfredo Pareto's garden were produced by naturally need to use the system more , and some less. So 20 % of the pods, and so on. then , how can those individual users and / or groups of users [0036 ] There are many kinds of systems in which power that are not using the application properly be identified ? laws apply : these tend to be complex systems with a large What is the absolute minimum acceptable level of usage in number of interacting parts . One such system is certainly an organization ? Can that be defined ? In order to address software usage. It fits the definition of a complex system . It these questions, an index was defined as 1 -GC of the data . is a system with many interacting parts: e.g., the number and type of users, the number and types of licenses, the setup of Power Laws the objects accessed by the users, the business processes , the amount of data entered by the users , the management methods and styles of the company or organization , etc. [0032 ] Empirically , a wide variety of systems follow what Every one of these interacting parts affects the usage patterns is known as “ power laws,” including man -made systems, of the software application . As such , it can be assumed that biological systems, or even physical phenomena. All these power laws hold true for software usage . Based on this systems tend to follow these power laws over a very large realization , a Pareto level usage ( that is, 80 % of application range of magnitudes . Take , for example , the sizes of earth usage is performed by 20 % of the people ) can then be used quakes, the frequencies of word usage in most languages, as a bare minimum acceptable usage pattern . In a preferred the distribution of craters on the moon , the frequencies of embodiment, the bare minimum acceptable level of usage is family names, wealth , criminal charges per convict — all of defined as such . these tend to follow power laws. [ 0033] There are situations in which one can be reasonably Preferred Embodiment certain that a particular statistical distribution is likely to [0037 ] Based on the Pareto principle , usage of any object, apply to the data in question . For example, the size of cities or usage by any group or user or organization , should have in any country follows a power law . If you rank all the cities 80 % of the activities done by at least 20 % of the people. If in the country, the largest by population is generally twice 80 % of the activity is done by less than 20 % of the people , the size of the second largest which is twice the size of the it indicates a likely problem . This should be an " easy " third largest and so on . Similarly, in the English language, benchmark to meet. Surprisingly, many organizations the most used word is used twice as often as the second most struggle to meet even this easy benchmark with their soft used which is used twice as often as the third most, and so ware application deployments. on . This kind of relation is called a power law and it was first [0038 ] To make the data easy to interpret, an index was discovered in 1949 by the linguist George Zipf. (Zipf's law defined that converts the usage pattern to an easy to under states that while only a few words are used very often , many stand number. A usage pattern of 80 % / 20 % (that is , 80 % of or most are used rarely , PM – 1/nº where P , is the frequency of the activity is done by 20 % of the users ) corresponds to an a word ranked nth and the exponent a is almost 1. This index of 40. A usage pattern of 70 % /30 % (that is, 70 % of the means that the second item occurs approximately 1/2 as often activity is done by 30 % of the users ) corresponds to an index as the first, and the third item 1/3 as often as the first, and so of 60. The higher the better. An index of less than 40 on . Although he originally intended it as a model for indicates a likely usage problem and the lower it goes the linguistics, Zipf later generalized his law to other disci more severe the problem . An index above 60 usually indi plines. In particular, he observed that the rank vs. frequency cates good , healthy , usage . distribution of individual incomes in a unified nation [0039 ] It is then possible to benchmark how much better approximates this law .) or worse software usage is when compared to the 80/20 benchmark . And , to do that, 1 -GC does the trick . When Pareto Principle 1 -GC = 0.4 , it means that usage pattern is 80/20 . At 1 -GC = 0 . 6 ,usage is now 70/30 (i.e., 70 % of the work is done by 30 % [0034 ] One example of a power law that particularly of the users ). This means that 1 -GC can serve as an occurs frequently in human -related phenomena is the Pareto indication of " quality of usage:" a minimum acceptable level principle (also known as the 80-20 rule , the law of the vital of 0.4 (which occurs at 80/20 ) with a target of trying to reach few , and the principle of factor sparsity ). The Parteto prin or exceed 0.6 (which occurs at 70/30 ). ciple states that, formany events, roughly 80 % of the effects [0040 ] In one example, different values for the Gini Coef come from 20 % of the causes. Mathematically , the 80-20 ficient correspond to different levels of inequality ( i.e., US 2019/0354912 A1 Nov. 21, 2019 different levels of the Pareto statement. One implementation analytic tool or software application utilizing the processes of an index includes modifying the GC value as follows: described above . This type of analysis enables benchmark ( 1 -GC )* 100 . This implementation changes the representa ing various functionalities of software deployment and iden tion such that perfect equality equals 100 rather than 0 so as tifying the areas that have most need for improvement. In to be a more readily understandable indication of equality one aspect, an interactive graphical user interface is pro rather than inequality . For ease of interpretation , one wants vided that enables easy viewing and manipulation of login to know what Pareto values correspond to what Gini values . information , usage information , and/or adoption information [0041 ] One skilled in the art would understand that there utilizing the benchmarking techniques described above. It are many differentways of calculating the GC , and there are should be appreciated that the information illustrated and many different definitions for different purposes. One described in reference to FIGS. 2-4 can also be generated in example is provided below . a static report for example . [0042 ] The Gini coefficient (or Gini ratio ) G is a summary [0046 ] FIGS . 2A and 2B illustrate example login and statistic of the Lorenz curve and a measure of inequality in usage information of a software application provided in a a population . The Gini coefficient is most easily calculated graphical user interface 200 (which includes displays or from unordered size data as the “ relative mean difference," screens 200a and 2006 ). By displaying the login and usage i.e., the mean of the difference between every possible pair information , comparison of certain portions of the data may of individuals, divided by the mean size u , be performed more effectively and efficiently , such that certain characteristics between the data may bemade imme ox; – xil diately apparent to a user of the system . For example, G = 2n2u comparing login frequency ( e.g., representative of customer adoption ) of a given software application , and an average of indicates text missing or illegiblewhen filed other usage, such as compiled by the system ( e.g. ,within the system or organization ), may be clearly illustrated at 205 . (Dixon et al. 1987, Damgaard and Weiner 2000 ). Alterna Interface 200 may provide login information and usage activity in easily understandable graphs, charts , etc., such as tively , if the data is ordered by increasing size of individuals , illustrated and in other forms. G is given by : [0047 ] FIGS. 3A and 3B illustrate example analytics of usage of a software application provided in a graphical user ( 20 -N 1) interface 300 (which includes displays or screens 300a and ?? 300b ).Using the benchmark information, determined via the indicates textmissing or illegiblewhen filed techniques described above, a scale can be developed , used , and displayed at 305 , 310 , and 315 based on data internal to an organization itself (e.g., not requiring external data to (Dixon et al. 1988 , Damgaard and Weiner 2000 ), correcting make the comparison against ), to identify different levels of the typographical error in the denominator given in the software adoption . The scale or distribution may include original paper (Dixon et al. 1987 ). multiple ranges of values that each correspond to , for [0043] The Gini coefficient ranges from a minimum value example, sub -adequate , adequate, and above-adequate soft of zero , when all individuals are equal, to a theoretical ware usage . The scale may be used to evaluate cross maximum of one in an infinite population in which every organizational tests , object adoption ( e.g., different features , individual except one has a size of zero . It has been shown functionality , options, etc., provided by the software), and that the sample Gini coefficients defined above need to be group metrics ( e.g., adoption by different groups within multiplied by n /( n - 1) in order to become unbiased estima organization analyzing software adoption ). Each individual tors for the population coefficients . item ( row ) may be evaluated separately, with a score illus [0044 ] The Gini coefficient for the Pareto distribution is trated in scale or chart 305 , 310 , 315 , at 320. As illustrated , then calculated for ( for axl) Frequency Rate (FR ) indicates the distribution of activities throughout the working days in a given month . A FR = 0 means all activities by a user are done on a single day in that month . A FR = 100 means activities are evenly distributed G =1--(['unyar)= 2a – 1 through the month . The described system may calculate Frequency Rates for groups or objects, as well as for the whole organization . A Pareto distribution ,where 20 % of the ( see Aaberge 2005) . According to the above formula , 80/20 users do 80 % of the activities, produces FR = 40 . Collabo corresponds to 0.6 , so (1 -GC )* 100 )= 40 , and 70/30 corre ration Rate (CR ) indicates the distribution of activities sponds to 0.4 so ( 1 -GC ) * 100 = 60 ). among the users of a given group . A CR = 0 means all activities in a group are done by a single user, a CR = 100 Benchmarking Software Application User Adoption means activities are evenly distributed among all the users in [0045 ] A lack of measurement leads to a lack ofmanage a group . A Pareto distribution , where 20 % of the users do ment. One cannot manage what one is not measuring . 80 % of the activities , produces CR = 40 According to the described systems and methods, a report [0048 ] FIGS. 4A -4C illustrate an expanded view of group that collates indexes for all aspects of usage can be generated adoption and usage information , as illustrated in FIG . 3B , in one document or view , screen , etc., to enable easily via graphical user interface 400 (which includes displays or identification of problem areas. FIGS . 2A , 2B , 3A , 3B , 4A , screens 400a , 400b , and 400c ). As illustrated in interface 4B , and 4C illustrate example reports generated by an 400 , login and usage information may be readily displayed US 2019/0354912 A1 Nov. 21, 2019 month by month , in a way the effectively enables compari terns that are at odds with good adoption . The described son of the data to identify trends or patterns . techniques may be used to find unusual and extreme behav [0049 ] The above interfaces help to illustrate the useful iors that are recurrent across all verticals . “ Good” behaviors ness of benchmarking software usage and adoption in an may be taught and become best practices while “bad ” organization according to the described techniques. The behaviors are contained or eliminated . In some cases , usage graphical user interfaces/reports described above are only data may be continuously crowdsourced from each organi given by way of example . Other arrangements of certain zation's activities in fine detail . Al or machine learning may data , and other graphical elements employed to convey the be applied to activities measured on objects by users pro information , for example , are contemplated herein . ducing behavioral patterns. In some cases , all users for all [0050 ] The described systems and methods are founded on activity using all objects may be compiled . Behaviors may an innovative recognition that this particular combination of then be detected using neural network classifiers. User relationships (the Gini coefficient and the Pareto principle ) attributes (e.g. profile, manager, role ) can be used to con produces a general purpose result, for all usage/activity / textualize behaviors and produce actionable outputs . other power law metrics in complex systems (not just [0055 ] FIG . 5 illustrates an example process 500 for software adoption, but any such power law style measure analyzing software usage of a group of users to determine ment). one or more practices for improvement, either on a group Using Benchmarking to Optimize Software Usage and level or individual level. In some cases , one or more Increase Performance operations of process 500 may be combined with process 100 described above . For example , in one aspect, one or [0051 ] In some aspects , after determining software usage more of operations 105 , 150, 155 , 160 , 165 , 170 , and/ or 175 as compared to the determined benchmark or benchmarks, may be preplaced or combined with one or more operations the analysis can be taken one step further . For example , 505-530 of process 500 to yield one process. As described usage may then be compared among users of the group to below , any reference to process 500 also refers to any analyze behaviors, to ultimately effect positive change in combination or process 100 and 500 , or process 100 alone. practice and procedure of an organization . In one example, [0056 ] Process 500 may begin at operation 505 , where the usage of a certain individual or group in a particular role usage information for a given software program may be may be dissected and analyzed . The usage may then be obtained , for example by and within an organization . Net, correlated with some performance metric or metrics (e.g., one or more benchmarks may be determined at operation customer satisfaction , resolution of tickets , etc.). This infor 510. In some cases, login information may first be obtained , mation can then be provided to inform decisions concerning and users divided in active and inactive users ( e.g., see changing practices of the group , asking specific users to operation 110 of process 100 ), in order to focus the results change their respective practices, etc. of process 500. At operation 515 , the benchmark data may [ 0052 ] The difference in adoption reflects , and often points be compared to actual usage data . In some cases, perfor to , differences in behavior. For instance , it is possible to mance information of the group may be obtained at opera differentiate between two users in the same micro -group tion 520. Performance data may include sales closed by an ( e.g. they are supposed to behave similarly ) according to associate , call request completion by a call center represen their frequency of usage of a certain functionality, such as tative, and so on , for a group of users. Next, the performance the number of case openings, or the number of updates to a information may be correlated with the usage data , at certain customer case . This has a direct effect on the effi operation 525. Operation 525 may include correlating cer ciency of each of the agents for each specific case handled , tain software usage patterns with adequate or satisfactory but over time also provides interesting trends relative to the performance data , sub -adequate , or even excellent perfor capabilities of each individual to yield behavior patterns. mance data . In some aspects , correlating may be performed [0053] This information , specifically because it can be via Al,machine learning , neural networks, or a combination generated internally, without access to information from or thereof. Next, at operation 530 , a scale or range of software concerning other organizations, provides valuable informa usage may be determined or generated based on the one or tion and insight into improving practices and performance of more benchmarks and the correlated performance data . The users of specific software applications, increasing efficiency, scale may indicate different levels or categories ofusage , for profits in business, etc. example , normal or satisfactory, sub -satisfactory , and excel [0054 ] In some cases, the techniques described herein can lent or above -satisfactory . Various examples of different help boost employee productivity and performance by using information and different ways to splay the information artificial intelligence (AI) or machine learning , to find best generated via processes 100 and/or 500 are described below . practices in an organization for every single activity and [0057] For example , FIGS . 6A and 6B represent usage every single group of people . The described techniques can information , and patterns gained therefrom , relating to lead provide effective tools for employee self-improvement and development by sales representatives or reps. In a sales morale . The described techniques can be used to find hidden environment, organization , etc., for example , sales rep may patterns that will increase sales, customer satisfaction and be expected to enter new leads as they happen . Using retention and ultimately repeatbusiness wins for companies. process 500 , reporting information may be analyzed con The described techniques can also be used to improve data cerning how often representative update leads. In an quality and adoption , and to implement best practices on the example , it was found that some reps updated their leads Salesforce platform or other similar platforms. The only once a month while the rest were updating as they described techniques can be used to promote excellence and collected them . By analyzing the frequency of lead collec accountability in the enterprise , find measurable best prac tion , it may be determined what is normal for a specific tices at the micro level, and show /teach every user how to group of sales reps and what is somehow different from such reach professional excellence . Some users may show pat normality. Among such reps there are some that will enter US 2019/0354912 A1 Nov. 21, 2019 their leads very seldom compared to the group . This is characteristics by case . This behavior has been found to be represented in FIG . 6A where each rectangle corresponds to organization -specific, leaving little or no room for external a sales rep . The surface of the rectangle is proportional to the benchmarking , but making the self -reference benchmarking number of leads in the period under scrutiny (the more the very valuable . leads the larger the rectangle ) and the color (shading) [0061] In some cases, process 500 may be used to predict represents the frequency (green = very frequent and red = very certain behavior within an organization ,based on correlation seldom ), according to the key provided in FIG . 6A . Fre between performance data and software usage data . quency is sorted from most frequent to least frequent and [0062 ] FIGS. 8A -8D illustrate software usage as corre each rep is assigned a color and a quantity of leads. lated to attrition in an organization . Process 500 may be [ 0058 ] FIG . 6B shows why a rep was characterized as utilized to determine the relationship between usage and having sub - normal reporting practices. This rep has pro attrition , as illustrated in FIGS. 8A -8D , and be further used duced 1,432 leads on one day and only a few leads on a few to predict when employees of a company will leave. The days during the month . As a comparison ,his/her colleagues analysis of software usage among computer educated are producing a few tens of leads every work day. Software employees can be used to predict critical behaviors such as utilizing process 500 may flag reps with such unusual employees who are planning to leave the company. Users behavior as they are not reporting on their leads with who plan to resign usually reduce their efforts while using sufficient timeliness. In one instance , correcting this behav such reporting tools as Salesforce . In the example illustrated ior was found to improve the lead management process, the in FIGS. 8A -8D , users who resigned show an 80 % drop in sales performance of the organization and even customer Salesforce activity in the last two weeks of employment if satisfaction . As a result , by determining abnormal behavior compared to the average of the previous 90 days . This of software in a certain circumstances, and correlating that pattern recurs for over 65 % of the users who left. with , in some cases, external evidence or information , [0063 ] By analyzing correlations between software usage performance of an organization may be improved . In some and employee actions, such as leaving a job , valuable insight cases , certain reps may manipulate reports to influence sales may be gained to address issues before they reach a tipping quotas. This type of behavior may be correlated with reps point. This may include salvaging the employee relationship logging leads at certain (e.g., recurring ) dates, reps having to prevent him or her from leaving , acting to protect the long “ silent periods” and short bursts of a high volume of company's data , etc. In some cases, the indicators of such activities, or specific periodic behaviors such as right before action may include a sudden reduction in quantity and or after the end of a quarter or in anticipation of a manage quality of logged data in a software application , or a ment review . In some cases , time and quantity may be recognizable shaped pattern of usage, as illustrated in FIGS . specific to an organization , such that individualized data and 8A -8D . In some cases , certain patterns may be false posi special rules for interpreting that data are most effective . tives, such as vacation, or company dependent behaviors This behavior has been found to be organization -specific , that may be normal for that company . In some cases, these leaving little or no room for external benchmarking , but types of behaviors or patterns may be accounted for in making the self-reference benchmarking very valuable. assessing correlations and sub -optimal performance or per [0059 ] In another example, as illustrated in FIGS. 7A and formance indicative of certain actions. In some cases, spe 7B , process 500 may be utilized in a call center scenario to cifically contact activities may be shown to increase on an increase performance, e.g., through increases in efficiency , employee's last week ofwork . This information may also be of call resolutions. Most call center agents can solve cases used to verify or increase the confidence in a prediction that assigned to them . Some have a surprisingly high ratio of an employee is about to leave the company. This behavior handovers (escalation ) to their colleagues , a bad habit that has been found to be organization -specific , leaving little or increases costs to the contact center and reduce customer no room for external benchmarking , but making the self satisfaction. In this example , process 500 may be used to reference benchmarking very valuable. analyze thehistory of Cases as recorded on Salesforce CRM [0064 ] Each of the processes, methods and algorithms and isolate agents with abnormal handover levels. The heat described in the preceding sectionsmay be embodied in , and map illustrated in FIG . 7A shows agents' handover ratio fully or partially automated by, code modules executed by over a certain period where each rectangle corresponds to an one or more computers or computer processors . The code agent. The surface of the rectangle is proportional to the and results of the processes may be stored on any type of number of cases handled in the period under scrutiny (the non -transitory computer- readable medium or computer stor more the cases the larger the rectangle ) and the color age device. (shading ) represents the handover ratio ( green = very low and [0065 ] The various features and processes described red = very high ), per the key illustrated in FIG . 7A . Handover above may be used independently of one another , ormay be ratio is sorted from higher to lower and each agent is combined in various ways. All possible combinations and assigned a color and a quantity of cases as measured by the subcombinations are intended to fall within the scope of this system . disclosure . In addition, certain methods or process blocks [0060 ] FIG . 7B illustrates why an agent was associated may be omitted in some implementations. The methods and with sub- normal call resolution performance . Process 500 processes described herein are also not limited to any may be used to flag reps with such unusual behavior as they particular sequence, and the blocks or states relating thereto are causing longer resolution time and frustration among the can be performed in other sequences that are appropriate . call center customers. Correcting this behavior may improve For example , described blocks or states may be performed the case management process and the performance of the in an order other than that specifically disclosed , or multiple organization. In some cases, assessing different behaviors blocks or states may be combined in a single block or state. may include determining behaviors among multiple call The example blocks or states may be performed in serial, in center locations, and level ofskill, and analyzing type and /or parallel or in some other manner. Blocks or states may be US 2019/0354912 A1 Nov. 21, 2019 added to or removed from the disclosed example embodi determining at least one benchmark for usage of the ments . The example systems and components described software application based on a power law and the at herein may be configured differently than described . For least one expected user , wherein the determining is example, elements may be added to , removed from or based on the usage data only internal to the organiza rearranged compared to the disclosed example embodi tion ; ments. comparing the obtained usage data with the at least one [0066 ] In some aspects, some or all of the systems and/or benchmark ; and techniques may be implemented in software , firmware, outputting results of the comparison . hardware, or a combination thereof, such as one or more 2. The non -transitory computer-readable medium of claim application -specific integrated circuits ( ASICs), standard 1 , wherein the software application comprises at least two integrated circuits, controllers (e.g., by executing appropri objects; ate instructions, and including microcontrollers and /or wherein the instructions for identifying the at least one embedded controllers ), field - programmable gate arrays ( FP expected user of the software application further com GAs), complex programmable logic devices (CPLDs ), etc. prise identifying the at least one expected user of the Some or all of the systems and processesmay also be stored software application for each of the at least two objects; ( e.g., as software instructions or structured data ) on a wherein the instruction for determining the at least one computer-readable medium , such as a hard disk , a memory, benchmark , further comprise determining the at least a network or a portable device . The described techniques one benchmark for each of the at least two objects ; and may be practiced using various computer hardware and wherein comparing the obtained usage data with the at configurations . least one benchmark is performed for each of the at [0067] Conditional language used herein , such as, among least two objects. others , “ can ," " could ,” “ might,” “may," “ e.g." and the like , 3. The non -transitory computer-readable medium ofclaim unless specifically stated otherwise, or otherwise understood 1, wherein the instructions, when executed by at least one within the context as used , is generally intended to convey processing device, enables the at least one processing device that certain embodiments include , while other embodiments to perform the operations of: do not include, certain features , elements , and /or steps . obtaining login information of users of the software Thus, such conditional language is not generally intended to application ; and wherein the instructions for identify imply that features, elements and /or steps are in any way ing the at least one expected user of the software required for one or more embodiments or that one or more application further comprise instructions for identify embodiments necessarily include logic for deciding , with or ing only users that have logged into the software without author input or prompting , whether these features , application based on the obtained login information . elements and /or steps are included or are to be performed in 4. The non -transitory computer-readable medium of claim any particular embodiment. The terms “ comprising," 1 , wherein the instructions, when executed by at least one “ including,” “ having” and the like are synonymous and are processing device , enables the at least one processing device used inclusively, in an open - ended fashion, and do not to perform the operations of: exclude additional elements, features , acts , operations and so forth . Also , the term “ or ” is used in its inclusive sense obtaining performance information of users of the soft (and not in its exclusive sense ) so that when used , for ware application ; example , to connect a list of elements, the term “or” means correlating the usage data with the performance informa one, some or all of the elements in the list . tion ; and [0068 ] While certain example embodiments have been outputting the results of the correlation . described , these embodiments have been presented by way 5. The non - transitory computer-readable medium of claim of example only and are not intended to limit the scope of 4 , wherein the instructions, when executed by at least one this disclosure. Thus, nothing in the foregoing description is processing device, enables the at least one processing device intended to imply that any particular feature , characteristic , to perform the operations of: step ,module or block is necessary or indispensable . Indeed , determining a scale of usage based on the at least one the novel methods and systems described herein may be benchmark and the results of the correlating the usage embodied in a variety of other forms; furthermore , various data with the performance information ; wherein out omissions, substitutions and changes in the form of the putting the results of the correlating further comprises methods and systems described herein may be made without outputting the results of the correlating relative to the departing from the spirit of the inventions disclosed herein . scale . The accompanying claims and their equivalents are intended 6. A non -transitory computer -readable medium on which to cover such forms ormodifications as would fall within the are stored instructions that, when executed by at least one scope of this disclosure . processing device , enables the at least one processing device What is claimed is : to perform the operations of: 1. A non -transitory computer-readable medium on which collecting customer usage data ; are stored instructions that, when executed by at least one fitting the customer usage data over a distribution; processing device , enables the at least one processing device comparing the customer usage data against the distribu to perform the operations of: tion ; and obtaining usage data of a software application of an organization ; determining , based on the data , at least one business value from the comparison. identifying at least one expected user of the software 7. The non -transitory computer-readable medium ofclaim application ; 6 , wherein the distribution comprises a Pareto distribution . US 2019/0354912 A1 Nov. 21, 2019 8. A non -transitory computer-readable medium on which are stored instructions that, when executed by at least one processing device, enables the at least one processing device to perform the operations of: collecting customer activity data ; creating a distribution ; modifying the distribution based on one or more assump tions based on the customer activity data ; fitting the modified distribution to the customer activity data ; and determining, based on the fitted distribution , at least one business value . 9. At least one computer-readable medium on which are stored instructions that, when executed by at least one processing device , enables the at least one processing device to perform the operations of: collecting data characterizing usage of a plurality of software applications hosted on a network by users of a set of at least one client device ; storing the collected data , wherein the collected data forms a collected data distribution ; comparing the collected data distribution with an assumed distribution ; determining, based on the comparison , at least one busi ness value for at least one of the plurality of software applications. 10. The non -transitory computer -readable medium of claim 9, wherein the determining is based upon utilizing an index of 1 - Gini Coefficient .