Advances in Animal and Veterinary Sciences
Research Article
Evaluation of Different Hygienic Practices Applied in Slaughterhouses and its effect on Beef Quality
Samah Elsaid Laban1, Mohamed Mahmoud Mashaly2, Aly Mahrous Aly3, Nouran Essam Maher4, Manal Moustafa Zaki1*
1Department of Veterinary Hygiene and Management, Faculty of Veterinary Medicine, Cairo University, Giza, 12211, Egypt; 2Department of food hygiene and control, Faculty of Veterinary Medicine, Cairo University Giza, 12211, Egypt; 3Faculty of African Postgraduate Studies, Cairo University Giza, 12613, Egypt; 4Faculty of Biotechnology, MSA University, 6th October City, Egypt.
Abstract | Meat constitutes the main source of animal protein for human consumers. Hygienic practices undertaken inside slaughterhouses affect the degree of meat quality and level of contamination. Sources of meat contamination inside slaughterhouses are; contaminated water, workers, and working environment including; contaminated equipment, surfaces, and aerosols. In this study, beef samples were collected from three slaughterhouses with different hygienic levels in Cairo, Egypt. Besides, water samples, environmental swabs, and workers’ hand swabs were collected from different sites of the same slaughterhouses. Microbiological analysis of samples was applied to determine total bacterial count (TBC) and total coliform count (TCC). A detailed questionnaire survey and a visual inspection checklist were applied to workers to assess the hygienic procedures and practices of the slaughterhouses under study. Results showed that total bacterial counts of beef samples were 3.1, 4.39, 4.43 log10 CFU/g for slaughterhouses 1, 2, and 3 respectively. Total coliform counts were nil for beef samples of slaughterhouses (1) and (2) and 4.43 log10 for that of the slaughterhouse (3). TBCs of water samples were 1, 1, and 1.3 log10/ml respectively while coliforms were negative for all samples. TBC of wall, floor, Knives, hooks, and workers’ hand’s swabs were highest in the slaughterhouse (3) followed by slaughterhouse (2) and lowest counts were found in the slaughterhouse (1). In conclusion, microbial counts of the analyzed samples together with answers to the questionnaire revealed that hygienic practices, level of education, training, and experience were better in the slaughterhouse (1) followed by slaughterhouse (2), and the most unhygienic practices and conditions were undertaken in the slaughterhouse (3) that was reflexed on the degree of beef contamination.
Keywords | Slaughterhouses, Beef quality, Contamination, Hygiene
Received | September 19, 2020; Accepted | December 08, 2020; Published | January 15, 2021
*Correspondence | Manal M. Zaki, Department of Veterinary Hygiene and Management, Faculty of Veterinary Medicine, Cairo University, Giza, 12211, Egypt; Email: [email protected]; [email protected]; [email protected]
Citation | Laban SE, Mashaly MM, Aly AM, Maher NE, Zaki MM (2021). Evaluation of different hygienic practices applied in slaughterhouses and its effect on beef quality. Adv. Anim. Vet. Sci. 9(3): 429-437.
DOI | http://dx.doi.org/10.17582/journal.aavs/2021/9.3.429.437
ISSN (Online) | 2307-8316; ISSN (Print) | 2309-3331
Copyright © 2021 Laban et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
INTRODUCTION
Meat is an important but expensive source of animal protein containing all essential amino acids, highly bioavailable iron, zinc, selenium and vitamins (Klurfed, 2015). The most common sources of meat are domesticated animal species such as cattle, pigs, poultry, buffaloes, sheep and goats. The meat derived from cattle is known as beef, it represents about 22% of world meat intake (FAO, 2020). This high nutritive value makes it an excellent medium for the growth of both spoilage and pathogenic bacteria (Kyayesimira et al., 2020). WHO, 2007 and Shonhiwa et al., 2017 considered contaminated meat to be one of the main sources of food-borne illnesses and death caused by the ingestion of agents (bacteria, microorganism cells, and toxins) that are not killed even after meat cooking. Examples of meat bacterial contaminants are; E. coli, Salmonella, Listeria monocytogenes, Clostridium perfringens, Clostridium botulinum and campylobacter species (Bersisa, 2019).
The function of an abattoir is the production of safe, healthy and wholesome halal meat under veterinary supervision and hygienic conditions to avoid exposure to a contaminated environment containing hazardous microorganisms (Rao et al., 2009; Tawaf, 2013; Serda et al., 2015; Zailani et al., 2016). Pathogenic microorganisms are normally found inside the digestive tract of healthy cattle or on the hides of live animals contaminated with feces, blood, hair, or bile. After slaughter, fecal matter and soil adherent to hides (average 4 kg) can be transferred to the surface of previously sterile meat during processing especially if performed on the floor in absence of a carcass suspension (hanging) system (Rana, 2014; Abdelwahed and Abdelgadir, 2019). As a result of careless evisceration, intestinal contents will spread onto the meat surface and begin to invade tissues so that the meat spoils quickly if the product is not handled correctly or transmitted to consumers causing diseases (Rahayu, 2006; Diyantoro and Wardhana, 2019). Also, there are many other sources of exterior microorganisms in abattoirs including; the use of contaminated water, poor worker handling, use of contaminated equipment (tables, knives, other equipment in cutting operations) and polluted air (Koffi-Nevry et al., 2011; Muhammad et al., 2012; Birhanu et al., 2017). Staff working with food must maintain a high degree of personal hygiene; wear suitable clean clothing, clean footwear, never eat or drink during work, don’t wear jewelry or watches etc. (Slobodan et al., 2017). Most workers lack awareness of the importance of personal hygiene that facilitates gross contamination of products by workers’ hands and clothes or allow flies, insects, rodents, dust and other specks of dirt to contact meat (Pradhan et al., 2018). The use of poor quality water source for carcass and equipment during slaughter and processing can aid contamination (Parvin et al., 2017). Other factors facilitating the spread of Food-borne diseases especially in developing countries are poor food handling and sanitation practices, inadequate food safety laws, weak regulatory systems, lack of financial resources to invest in safer equipment, and lack of education for food handlers (WHO, 2004; Kyayesimira et al., 2020). All the previously mentioned contamination factors result in changes in the chemical composition and biological characteristics leading to spoilage of meat or spread infections in the community (Olaoye and Nilude, 2010).
To improve the meat quality, the abattoir should be approved to provide the requirements of the responsible authority in the country that include; suitable electricity supply, a water supply with sufficient quantity and quality, the abattoir layout must allow prevention of cross-contamination and enough separation of different activities, presence of effluent treatment system and hygienic waste disposal. The construction of the building and the equipment used in abattoirs must allow easy cleaning and sanitation and be properly maintained. Also, abattoir should implement Good Slaughtering Practices (GSP) to ensure the safety of food at all stages in the food chain (Codex Alimentarius Commission, 2004). The GSP stages include the cleanliness of production facilities, using clean water during the processing and implementation of sanitation programs (Harris and Jeff, 2003; Diyantoro and Wardhana, 2019). Safety and quality can be estimated based on indicator microorganism counts like mesophilic aerobes and total coliforms (Birhanu et al., 2017). The former counts provide an estimation of the total microbial population; their high levels are usually associated with low quality and reduced meat shelf life. The later counts allow verification of hygiene-related problems and contamination of fecal origin (Barros et al., 2007).
Therefore, the objective of this study was to investigate the relationship between beef quality and hygienic practices applied in three slaughterhouses (abattoirs) with different hygienic levels in Egypt.
MATERIALS AND METHODS
Questionnaire
A structured questionnaire was prepared to assess the knowledge of 45 workers from three different slaughterhouses (15 per each slaughterhouse) regarding the hygienic and sanitary practices during slaughter and processing of beef. The aim was to collect information about Socio-demographic characteristics, knowledge, perception, trainings, and experience in the processing of beef and slaughterhouse hygiene practices. Also a checklist of slaughterhouse hygiene, workers’ hygiene, and maintenance of the facilities was carried out.
Sample collection
Sampling and microbiological examination were applied according to Pradhan et al., 2018 with some modifications.
Swabs
A total of 91 swabs were collected from 3 different slaughterhouses in Cairo-Egypt (30 swabs per each slaughterhouse and one control negative swab). 6 swab samples were obtained from each type of beef contact surfaces and equipment that include; floor, walls, knives, hooks and workers’ hands. Briefly, an area of 25 cm2 (5cm X 5cm) was marked on the sampling surface with a sterile frame. Sterile cotton-tipped swabs were moistened with sterile normal saline then rubbed against the sampling sites for around 30 seconds and then transferred to an individual screw-capped test tube containing 5 ml of sterile maintenance medium (0.85% NaCl and 0.1% peptone), samples were properly labeled.
Water samples
Five water samples were collected from each one of the three different slaughterhouses water sources used for carcass cleaning and processing, where 100 ml sterilized plastic cups were used for the collection of each water sample.
Beef samples
Beef samples (n=36) each of 100 g (50 g from fore Shank and 50g from opposite hind Shank) were collected from the 3 slaughterhouses (12 sample per each slaughterhouse) representing 100% of carcasses. The sampling was performed following the completion of carcass dressing before the commencement of chilling. Samples were collected in sterile cups using a sterile knife.
All swab containing tubes, cups of water, and beef samples were transferred to the laboratory in an icebox at 0–5 oC (Australian Standard, 2018) where they were prepared and examined within 12 hours.
Sample preparation
The tubes containing swabs were vortexed for 30 seconds to ensure uniform distribution of microorganisms in maintenance media. Beef samples grinding was applied separately for each slaughterhouse samples then 10 g were collected and homogenized using a sterile blender in a sterile jar containing 90 ml sterile normal saline according to ICMSF (2002).
Microbiological examination
Determination of total bacterial (TBC) and total coliform (TCC) counts
In the laboratory, under complete aseptic conditions; ten-fold serial dilutions of all the original diluents containing the swabs, water and beef samples were made using test tubes containing 9 ml sterile saline solution (0.85% NaCl). Using the spread plate method, 0.1 ml of the original sample and each dilution was transferred on duplicates of solidified previously prepared plate count agar and MacConkey agar plates for determination of total bacterial count (TBC) and total coliform bacteria count (TCC), respectively. The diluted samples were spread as quickly as possible on the surface of plates using a sterile glass spreader. The inoculated plates were incubated under aerobic conditions at 37°C for 24 hrs. After incubation, plates with countable colonies (30-300 numbers) were counted using the illuminated colony counter. The total count obtained was multiplied with the dilution factor according to ISO (2009) and expressed as a colony-forming unit (CFU). The microbiological data were expressed in CFU/cm2, CFU/ml and CFU/g in the case of swabs, water, and beef samples, respectively.
Determination of the most probable number of water
The multiple-tube fermentation technique for most probable number (MPN) was carried out to determine the number of total coliform bacteria as described in the Standard Methods for the Examination of Water and Wastewater by Eaton et al., 2005. The numbers of positive findings were enumerated according to (MPN tables) to determine bacterial counts.
RESULTS AND DISCUSSION
Microbial analysis
Meat contamination and cross-contamination occur mostly due to inadequate hygienic conditions and improper handling in slaughterhouses (Schlegelova et al., 2004; Koo et al., 2013). In our study biological indicators (TBC and TCC) were determined in beef samples obtained from a slaughterhouse (1), (2), or (3). Environmental swabs were collected and workers’ hands were swabbed and examined to detect the relationship between the degree of contamination and hygienic measures applied inside the slaughterhouse during beef processing. Results of microbiological examination of beef samples present in the Table 1 and Figure 1 showed that the total bacterial count was lowest in beef samples obtained from the slaughterhouse (1) followed by slaughterhouse (2) and highest counts were obtained from samples of the slaughterhouse (3) and counts were (1.3×103), (2.5×104), (2.7×104) with 3.1, 4.39, 4.43 log10 respectively. However, the highest total coliform count of beef samples was obtained from slaughterhouse 3 with a count of (2.7×104) and a log10 of4.43. According to the Center of Food Safety (2014), The satisfactory limit of bacteria and coliform should be less than 103 and a range of 103−105 is considered as the borderline limit however count more than 105 is considered unacceptable. Also, WHO, 2007, agreed that the limit of 105 CFU/cm2 or 5.0 log10 CFU/cm2 of total plate count on meat is the maximum limit. If the bacterial count exceeds the above standard in fresh meat, then the meat is not acceptable and this indicates alarm signals on meat hygiene. Results of beef samples under study were more than 103 but lower than 105 so considered as the borderline limit.
Figure 1: log10 CFU/g of total bacterial counts (TBC) in water and beef samples obtained from the examined slaughterhouses.
Table 1: log10 CFU/g of Total bacterial counts (TBC), Most Probable Number of coliform bacteria (MPN), and total coliform bacteria counts (TCC) of the examined water and beef samples.
| Slaughterhouse | Water | Beef | ||
|
TBC log10 (CFU/ml) |
MPN of coliform bacteria log10(CFU/100ml) |
TBC log10 (CFU/g) |
TCC log10 (CFU/g) |
|
| Slaughterhouse 1 | 1 | 0 |
3.1(1.3×103 ) |
0 |
| Slaughterhouse 2 | 1 | 0 |
4.39(2.5×104) |
0 |
| Slaughterhouse 3 | 1.3 | 0 |
4.43(2.7×104 ) |
4.43(2.7×104 ) |
In the three examined slaughterhouses, the slaughtering process was applied on the ground. Method of slaughtering and meat processing (ground or vertical rail dressing) has a great role in determining the degree of contamination. Bovine slaughtering on the floor is considered a chief source of meat contamination by a huge number of bacteria that may have transmitted from animals to the environment and consequently to the production line to contaminate other carcasses. Results of swabs in Table 2 and Figures 2 and 3 showed that the highest counts of bacteria and coliforms were obtained from floor swabs this can be explained by the presence of accumulated blood, animals’ internal organs and contaminated water on the floor. The obtained floor counts were higher in slaughterhouses 1 and 2 than in slaughterhouse 3 is proportional to the numbers of animals slaughtered per day in each slaughterhouse. In a study applied by lines Bakhtiary et al., 2016 on two types of processing lines they found that sheep slaughtering and processing on the ground facilitated a high level of meat contamination. However, in the same study, in cattle slaughterhouse, vertical rail dressing improved hygienic practice by reducing carcass contact with operators, equipment and other carcasses. Others concluded that pre-slaughter conditions and evisceration process can facilitate carcass contamination by fecal contamination of hides and spread of gut bacteria and the possibility of high-risk foodborne pathogens that can spread contamination on meat, floor and equipment surfaces. These bacteria will lead to rapid spoilage within the first hours of storage or during prolonged storage under room temperatures (Lavilla et al. 2013).
Table 2: log10 of total bacterial count (TBC) and total coliform count (TCC) of swabs collected from different slaughterhouse surfaces and equipment.
| Sampling site | Slaughterhouse 1 | Slaughterhouse 2 | Slaughterhouse 3 | |||
| TBC | TCC | TBC | TCC | TBC | TCC | |
|
Wall |
2.3 | 0 | 2.5 | 0 | 3.7 | 0 |
| Knife | 3.15 | 0 | 3.67 | 1.7 | 4 | 0 |
| Workers’ hands | 3.9 | 2.9 | 3.5 | 0 | 4.4 | 1.2 |
| Floor | 5.4 | 5.2 | 5.4 | 2.9 | 4.1 | 3.2 |
| Hook | 3.8 | 1.7 | 2.8 | 0 | 4.2 | 0 |
Figure 2: log10 of total bacterial count (TBC) of swabs collected from different slaughterhouse surfaces and equipment.
Figure 3: log 10 of the total coliform count (TCC) of swabs collected from different slaughterhouse surfaces and equipment.
Results in Tables 2 and 3 and Figures 2-6 showed that swabs obtained from workers’ hands had high bacterial counts and the highest counts were obtained from slaughterhouse 3. Gordon-Davis, 1998; Hardstaff et al., 2018, stated that the major risks of food contamination, and a substantial number of foodborne illnesses can originate from poor food handling practices of food workers and they concluded that disease-causing micro-organisms exist in the food workers body can consequently transport from them to the food during the handling process. According to questionnaire answers; workers of this slaughterhouse didn’t receive training about good practice in the food industry, work safety nor work hygiene. In slaughterhouses 1 and 2; 40% and 20% of workers respectively received previous training in the previously mentioned sectors. The level of experience was 3-5 years in 93% and >5 in only 7% of workers against 3-5 years’ experience in 12% and >5 in 88% in workers of the slaughterhouse (1) and 3-5 years’ experience in 87% and >5 in 13% workers for the slaughterhouse (2). The level of education, and training of food handlers in food safety constitutes effective strategies for preventing foodborne diseases (Adams and Moss, 1997; Rossi et al., 2017).
In table number (2); variable degrees of contamination were recovered from different swabs and the highest counts were on surfaces, equipment, and knives of the slaughterhouse (3). Workers of the same slaughterhouse according to the questionnaire answers didn’t use different knives for handling meat from different animal species. This agrees with the results of Bakhtiary et al., 2016 who concluded that contamination in cutting and deboning areas may increase by knives, contact surfaces and hands of workers. They added that cleaning the knives, surfaces, and trays with hot water in addition to training the workers has a great effect on the reduction of cross-contamination. Also, agree with Soeparno, 2009; Endale and Hailay, 2013 who stated that anything that contact meat directly or indirectly, can be a source of microbial contamination. The level of training affected the decision of workers concerning dealing with meat portions with clear signs of damage with different procedures as presented in Figure 5.
Carcass dressing water constitutes the main source of meat contamination. Results of water samples present in Table 1 and Figure 1 declared that TBC was higher in the water sample of the slaughterhouse (3) than 1 and 2 however the MPN of coliforms in the three slaughterhouses water samples was zero. Bersisa et al., 2019 mentioned that water used in a slaughterhouse can contaminate meat during washing. They recommended using water with drinking water standards for cleaning and meat processing in the abattoir. The average total bacterial counts of water samples in this study were 1, 1, and 1.3 log10 CFU/ml. These values are lower than those mentioned in the report of Tarwate et al., 1993 who reported a mean value of 2.1 log10 CFU/ml from water in the abattoir, and the report of Pius, 2013 who reported a mean value of 4.3 log10 CFU/ ml in Ibadan, Nigeria.
Figure 6: Results of daily storage temperature check procedure for individual slaughterhouse workers.
Figure 7: Results of procedures undertaken by the slaughterhouse workers concerning beef portions with clear signs of damage.
According to answers of the questionnaire in the slaughterhouse (1), 67% of workers answered that they practice cleaning and disinfection during work against only 3% in the slaughterhouse (2) and zero% in the slaughterhouse (3) while all the rest answers said that they practice cleaning and disinfection at the end of the working day. Also, cleaning was applied with water and soap without using any type of disinfectant. This disagrees with Sander et al., 2002 who recommended using disinfectants that are effective and efficient against a wide variety of micro-organisms and doesn’t tent the meat and its marketability for routine slaughterhouse disinfection. Results coincide with Dixon et al., 1991 who concluded that, inefficient sanitation and improper disinfection program in a slaughterhouse increase level of carcass contamination. Also, concur with Rahkio and Korkeala, 1996 in that
Table 3: Answers to questionnaire and checklist questions.
| Question | Answer | ||
| Slaughterhouse 1 | Slaughterhouse 2 | Slaughterhouse 3 | |
| 1-Age (years) |
24-30 (30%) 31-45 (49%) >45(21%) |
24-30 (49%) 31-45 (31%) >45(20%) |
24-30 (56%) 31-45 (34%) >45(10%) |
| 2- Education level |
Primary/ Secondary education (60%) High school education (40%) |
Primary/Secondary education (80 %) High school education (20%) |
Primary/Secondary education (60%) High school education (20%) University education (20%) |
| 3-Relevant experience (years) |
3-5 (12%) >5 (88%) |
3-5 (87%) >5 (13%) |
3-5 (93%) >5 (7%) |
| 4-Training in good practice in food industry |
Yes (40%) No (60%) |
Yes (20%) No (80%) |
No (100%)
|
| 5-Training in work safety and work hygiene |
Yes (40%) No (60%) |
Yes (20%) No (80%) |
No (100%)
|
| 6-Most recent training |
Last year (2%) More than 3 years ago (38%) No training (60%) |
Last year (0%) More than 3 years ago (20%) No training (80%) |
No training |
|
7-Identification of common symptoms of foodborne diseases |
Body pain (13%) Abdominal pain, fever, vomit, and diarrhea (87%) |
Body pain (26%) Abdominal pain, fever, vomit, and diarrhea (74%) |
Body pain (34%) Abdominal pain, fever, vomit, and diarrhea (66%) |
| 8-Use of different knives for handling beef from different animal species |
Yes (34%) No (66%)
|
Yes (3%) No (97%)
|
No (100%)
|
| 9-Meat preparation at the same workbench |
Yes (45%) No (55%) |
Yes (57%) No (43%) |
Yes (93%) No (7%) |
| 10-Type of water | Public water supply (100%) | Public water supply (100%) | Public water supply (100%) |
| 11-Different types of fresh beef and/or beef products placed in the same bag |
Yes (4%) No (96%)
|
Yes (31%) No (69%)
|
Yes (33%) No (67%)
|
| 12-Procedure to remove beef portions with clear signs of damage |
Remove the meat portion for general waste (80%) As directed by the responsible person (20%) |
Remove the meat portion for general waste (60%) As directed by the responsible person (40%) |
Remove the meat portion for general waste (20%) As directed by the responsible person (80%) |
| 13-Hygiene plan |
Yes (98%) No (2%) |
Yes (33%) No (67%) |
Yes (27%) No (73%) |
| 14-Types of disinfectants be used | Soap and water (100%) | Soap and water (100%) | Soap and water (100%) |
| 15-Time of cleaning and disinfection |
After the end of the work (33%) During the work (67%) |
After the end of the work (97%) During the work (3%) |
After the end of the work (100%)
|
| 16-Number of times cleaning and disinfection the slaughterhouse |
Daily (100%)
|
Daily (100%) |
Daily (100%)
|
| Daily check of storage temperature |
Yes (100%) No (0%) |
Yes (34%) No (66%) |
Yes (31%) No (69%) |
applying efficient hygienic practices such as regular disinfection of working tools and workers’ hands is important in reducing the microbiological contamination of carcasses. Also, they mentioned that it is important to create awareness among workers about proper hygienic measures and regular sanitation during meat processing besides gaining a sufficient level of education, training, and experience, this coincides with our questionnaire results that explained in Figures 4-7.
According to the questionnaire applied the collected answers cleared that workers of slaughterhouse 1 apply daily check of storage temperature when compared to the other two slaughterhouses. Temperature is one of the major factors affecting microbiological growth. During carcass storage, low temperatures are critical to prevent microbial spoilage. The prime purpose of a meat refrigeration process is to reduce the temperature of the meat to a value below which the rate of bacterial growth is either severely slowed (chilling) or stopped (freezing). Red meat carcass-chilling systems depend on refrigerated air as the cooling medium using large insulated rooms with hanging rails for the meat carcasses or sides (James and James, 2009). The minimum growth temperature is a function of the particular organism and also the type of food or growth media that is used for the incubation. According to James, 2002, risks to food safety are reduced if the meat is maintained below 5ºC. From the economic point of view, rapid rates of chilling can reduce the amount of drip loss to half. So low temperatures of preservation not only keep the microbial quality but also leads to a reduction in weight loss, which results in a higher yield of saleable material (Aidani et al., 2014). This can discuss the elevated microbial quality of slaughterhouse 1 beef samples rather than the other two samples.
CONCLUSION AND RECOMMENDATIONS
Results obtained from this study showed that there were different levels of microbial loads in the collected samples. Also, different levels of worker training, education and experience were found in the examined slaughterhouses. The resulted total bacterial counts and coliform counts from the tested samples indicate relatively poor beef quality. However, the satisfactory limit of bacteria and coliform should be less than 103, the results of our samples belong to the range of 103−105 which is considered as the borderline limit. So, it can spoil rapidly or may act as a potential source of food-borne infection. Many factors favored contamination such as poor hygienic and sanitation procedures conducted at the slaughterhouses especially slaughterhouse number (3) that was characterized by lack of training, low education and experience levels of workers. Consequently, it is important to create awareness among workers about proper hygienic measures and regular sanitation during meat processing besides gaining a sufficient level of training and experience.
ACKNOWLEDGMENTS
The authors gratefully thank the managers of the examined abattoirs for their help in the examination of the slaughtered animals and providing the samples under the study. Special thanks also go to the abattoir staff and casual workers for their cooperation during data collection. Also, the authors wish to thank the Department of Veterinary Hygiene and Management, Faculty of Veterinary Medicine, Cairo University for the cooperation during doing the practical parts in their laboratory.
Author’s Contribution
All authors contributed equally in creating the article.
Conflict of interest
The authors have declared no conflict of interest.
REFERENCES
