PART I: SECONDARY DATA REPORT
1.1 Objectives and Methodology
The Library Services and Technology Act (LSTA) replaced in 1996 the long standing Library Services and Construction Act (LSCA). LSTA "places greater emphasis on programs that support the use of technology in libraries and promote cooperation efforts between libraries, including school, academic and research libraries," according to the Texas State Library and Archives Commission's strategic plan for 2001-2005: Beyond 2000: Books, Bytes, and Beginnings. The LSTA set six priorities. These include:
- Establish or enhance electronic linkages between and among libraries.
- Link libraries with educational, social, or information networks.
- Assist libraries in accessing information through electronic networks.
- Encourage libraries in different areas and encourage different types of libraries to establish consortia and share resources.
- Pay costs for libraries to acquire or share computer systems and communication technologies.
- Target library and information services to persons having difficulty using a library and to underserved urban and rural communities.
LSTA funds support services to a diverse population and to all types of libraries. LSTA also set financial guidelines regarding program support and the amount dedicated to direct service versus administrative costs. According to LSTA, federal funds can support only 66 percent of program costs and only up to four percent of federal funds can be used for administrative costs; 96 percent have to be used for direct projects or grants.
The Texas State Library and Archives Commission's (TSLAC) general goal is "to establish, extend, and improve library services in all areas of the state.to improve the availability and delivery of library and information services to all Texans," under LSTA and the Texas Library Systems Act (LSA). TSLAC has set four strategies to accomplish this goal:
- Share library resources among libraries statewide through interlibrary loans, electronic library services, state publications, and clearinghouse.
- Support the Texas Library System by funding programs to improve regional groups of libraries, including purchasing of materials and equipment, technical assistance and training, and other cooperation projects.
- Provide services and grants to aid the development of local libraries, including direct grants, continuing education and consulting, Texas Reading Club, and other services.
- Provide direct library services by mail to Texas citizens with disabilities from a centralized collection of large-print, Braille, and recorded books.
In order to achieve its goal, TSLAC has set priorities and strategies, which are consistent with the Library Services and Technology Act through a series of eight projects and programs. These include:
- Texas Library Systems
- Technical Assistance Negotiated Grants
- Special Projects Grants
- Library Cooperation Grants
- Continuing Education and Consulting
- Public Library Establishment Grants
- TexNet Interlibrary Loan Program
- Texas State Electronic Library Program (currently part of TexShare)
The TSLAC engages in extensive monitoring, documentation, and self-evaluation of its activities and programs. These include ongoing communication with and visits to subgrantees, annual audits, surveys of providers and customers (patrons), and periodic statistical and narrative reports by the subgrantees, performance reviews, and evaluations.
The purpose of this project was to evaluate the overall effectiveness of the current State Plan for the Library Services and Technology Act and its program activities over a four- year period: 1998 to 2001. The evaluation used different methodologies such as TSLAC data and reports (secondary data), surveys of coordinators of the Texas Library Systems, directors of member libraries, directors of Special Projects grant recipients and a sample of individuals who received services from these grants. In addition, visits were conducted to two Special Projects Grant recipients and to three Library Systems. These methodologies resulted in the in-depth evaluation of three of the eight LSTA programs. The LSTA-funded programs evaluated in-depth included the Texas Library Systems program, the Special Projects Grant program, and the Technical Assistance Negotiated Grants program.
Specifically, the objectives of the evaluation of the LSTA-funded programs based on secondary data were to:
- Examine activities for each program performed in 1998, 1999, 2000, and 2001.
- Evaluate programs by considering:
- allocation of funds
- libraries served
- impact of program
- Address how well programs promoted access to information resources in all types of libraries and stimulated excellence.
- Document major accomplishments and successful practices. � Identify areas for improvement.
- Make recommendations about the grant process and areas for improvement.
- Show levels of success in achieving the LSTA plan goals.
1.2 Performance Measures
The eight LSTA programs provide the Texas State Library and Archives Commission (TSLAC) with data on a quarterly and annual basis both in narrative form and through statistics corresponding to the performance measures set out by TSLAC. The TSLAC developed a set of broader performance measures that are applicable across the eight programs. The performance measures typically are output measures as shown in the following table.
Key Performance Measures
Texas Library Systems
Library Establishment Grant
Library Cooperation Grant
Technical Assistance Negotiated Grant
Continuing education and Consulting
TexNet Interlibrary Loan
Texas State Electronic Library/ TexShare Online Databases
Overall, the LSTA-funded programs show a high level of activity on the part of Library Systems and member libraries. The activities performed are consistent with the priorities set by LSTA.
The performance data the Library Systems and other grantees provide to TSLAC also raise several issues.
The data that was reported by the eight LSTA programs on the key performance measures show a high level of activity on the part of the programs in all performance areas. The actual performance is compared with a set target. In nearly all LSTA grant programs and in all areas of performance, the performance of the program or Library System is typically far from the target; either far above or far below the +/- five percent deviation the TSLAC "accepts." This pattern also holds across the four evaluation years. That is, grant programs or Library Systems do not use their actual performance during a specific year to adjust their target for the following year. A review of targets by year shows little correlation to actual performance in the previous year.
Targets are considered a useful management tool, helping organizations to set a criterion against which to judge their performance. Targets are also useful in guiding organizations in resource allocation. Meeting set targets or coming close to them allows organizations to operate efficiently. While Library Systems and other grant programs use targets, it is clear that they do not follow the targets they set nor do they base the setting of targets in the following year on their actual performance and their ability to increase or maintain this level of performance. One area in which TSLAC can assist Library Systems and grant programs is in developing a methodology or guidelines for setting targets and refining targets based on actual performance. The TSLAC should also train Library Systems and other grant programs to use targets as a management tool both in planning, resource allocation, and monitoring.
One objective of this report was to create a longitudinal picture of the LSTA-funded programs. The data provided to TSLAC by LSTA-funded programs does not show any clear longitudinal patterns. That is, the performance from year to year or across several years by Library Systems and LSTA funded programs is inconsistent. It is inconsistent by individual programs as well as by program categories. This inconsistency shows up across all performance measures. The lack of patterns of program performance across several years suggests that operations are independent from year to year, and are not building on what was accomplished before or taking into consideration how performance can be improved. It is important for TSLAC to understand this lack of performance continuity and to guide the grant programs and the Library Systems in building on previous year accomplishments and operations.
The performance measures show a great deal of activity in several areas but do not provide necessarily the type of information that will help the TSLAC and individual organizations assess the extent to which they serve their population. For example, the performance measures tell us how many individuals received project sponsored services. However, this count is duplicative; that is, it does not count unique persons but rather the number of contacts or "service units". It is not clear, therefore, how many "unique" people actually received services, since some of these received multiple services or had multiple service contacts. Current data also do not allow us to determine if the number of unique individuals grew from year to year or if the library or the program served the same number of individuals or a smaller number of individuals but had a greater number of contacts with them.
Similarly, the performance measures in several of the grant programs address the number of librarians trained or assisted. Combining training and assistance obscures this measure because it combines two activities that have different operational implications. Having a trained librarian force should imply and result in a lower need for assistance. Determining the level of assistance that is needed, has in turn, implications for the type of training that is needed. The ultimate objective for program and organizational managers and for the TSLAC is to know the number of libraries that had one or more librarians trained. As this performance measure counts "training participants" rather than unique individuals and does not associate either with a unique library, the extent of training coverage remains unclear. This performance measure does not tell us whether the training efforts of the Library System or another grant program have included all libraries, or what types of libraries (size, location) were included.
The number of assistance contacts is a measure that Library Systems and grant programs can use to determine how much training and what type of training are needed in order to reduce the number of assistance contacts. In its current form, it does not give us this indication. It also does not reveal how many libraries are associated with these assistance requests.
Productivity and Efficiency
Productivity and efficiency are two aspects of program effectiveness (impact is a third aspect). Productivity and efficiency are typically calculated by dividing the activity volume by number of staff associated with this activity, the $ per service unit, or the portion (percent) of the activity that the staff managed to perform or to complete. The TexNet ILL program has used these measures well. All LSTA-funded programs can benefit from using productivity and efficiency measures. However, it is important that these be based on the amount of funds and personnel allocated and assigned to the activities being measured. For example, to determine the productivity of consulting activities that Library Systems provide, it is important to identify the funds that the Systems allocated to consulting and the staff that provided these services before making any calculations.
Having appropriate productivity and efficiency data the TSLAC can develop performance models in different activity areas and define productivity and efficiency targets; thus giving the Library Systems a standard against which to compare their performance.
Library Systems allocate resources based on the needs and priorities expressed by member libraries. Using this method of resource allocation is certainly responsive to the Systems' members. The question is whether this resource allocation method best serves the interests of the Library Systems and the TSLAC in a time of transition and rapid change and whether Library Systems and individual libraries are moving fast enough to take advantage of new developments. For example, an analysis of funding allocation in FY 2000 by Library Systems shows that Library Systems, on average, allocated 41 percent of their LSTA funds to collection development (a traditional area) and only 11 percent to automation and networked resources. In fact, the allocation of funds to this area varied significantly across Library Systems with some Systems allocating less than two percent to this area, and six out of ten Systems allocating less than ten percent to this area. One strategy that the TSLAC can use in moving Library Systems and thereby individual libraries to increase their electronic and networked resources and electronic resource use is by formulating state priorities and providing guidelines to Library Systems on how to allocate funds.
Libraries and Library Systems collect a large amount of data and report it to the TSLAC, responding to state and federal data requirements. The collection and reporting of the data must pose a considerable burden on staff and time resources. Yet, an analysis of the data reported shows that the majority of the data are not "actionable." That is, these data do not help libraries and Library Systems determine whether they are "doing a good job" or whether they provide services in an efficient manner. Moreover, the data collected do not represent the new reality of the library as a technology-heavy organization; it is more consistent with the pre-technology era.
The TSLAC jointly with Library Systems and library directors should undertake a major data review project to identify a small set of data items with which libraries, Library Systems and TSLAC can determine whether the data reflects the services they provide, and whether the libraries provide these services efficiently. Data items should also enable libraries, Library Systems and TSLAC to determine whether the services have the intended impact, and whether the data is action-based. That is, whether it will help the library, Library System or TSLAC take action in response to the situation.
Going Beyond Output Measures
The Library Systems engage in a high volume of activities. However, the data they provide does not reveal the impact these services have on the recipients and, in turn, on the Library System, on libraries or on the LSTA-funded program. The absence of impact measures limits the evaluation of LSTA-funded programs and activities and constitutes a gap that the TSLAC should fill. For example, the TANG program performance measures address the number of library staff trained, number of library staff assisted, and instructional materials provided. However, these performance measures do not indicate the extent to which these services increased libraries' technological self-sufficiency, reduced the number of "basic" technology calls for assistance, or increased use of electronic databases by patrons. Developing impact measures for the different programs requires a higher level of sophistication on the part of both the TSLAC and the grant recipients. The development of impact criteria also requires distinctions that these organizations did not previously make in their data reporting. It may be necessary to distinguish between "basic" and "advanced" training or between "basic" and more advanced assistance calls. However, utilizing impact measures would present significant benefits for the TSLAC, the Library Systems, and the LSTA grant programs.
Using Performance Measures As Management Tools
It is unclear at present to what extent Library Systems, member libraries and other LSTA-funded programs use the performance measures as tools to analyze their operations and achievements and improve their programs. The data appear to indicate that Library Systems and other grant programs do not use the target strategy as a management tool; they look at the set target as a TLAC or LSTA requirement. These organizations also do not seem to look at their performance over time to adjust their performance. The TSLAC needs to train and assist these organizations in the use of these performance data as management tools and to develop a longitudinal perspective of their operations rather than consider each year as an independent event.
II. TEXAS LIBRARY SYSTEMS
The Texas State Library and Archives Commission (TSLAC) created ten Library Systems. The Library Systems vary in the number of their member libraries and in the population size they serve. The Library Systems include:
- The Alamo Area Library System (AALS), headquartered in San Antonio, with 44 member libraries and one non-member.
- The Big Country Library System (BCLS), headquartered in Abilene, with 37 member and five non-member libraries.
- The Central Texas Library System (CTLS), headquartered in Austin, with 67 member and four non-member libraries.
- The Houston Area Library System (HALS), headquartered in Houston, with 67 member and two non-member libraries. � The Northeast Texas Library System (NETLS), headquartered in Garland, with 98 member and eight non-member libraries.
- The North Texas Regional Library System (NTRLS), headquartered in Fort Worth, with 71 member and one non-member libraries.
- The South Texas Library System (STLS), headquartered in Corpus Christi, with 52 member and two non-member libraries.
- The Texas Panhandle Library System (TPLS), headquartered in Amarillo, with 28 member and three non-member libraries.
- The Texas Trans-Pecos Library System (TTPLS), headquartered in El Paso, with 15 member libraries and one non-member.
- The West Texas Library System (WTLS), headquartered in Lubbock, with 34 member and one non-member library.